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I.   INTRODUCTION 

A .   BACKGROUND 

As  seen  in  several  of  the  recent,  wars  and  conflicts, 
speed  and  timing  are  crucial  in  modern  warfare.  In  the 
Falklands  War,  the  lack  of  time  available  to  react  to  a 
threat  caused  the  loss  of  HMS  SHEFFIELD.  The  HMS  SHEFFIELD 
was  sunk  by  fires  that  could  not  be  brought  under  control  as 
a  result  of  a  strike  by  an  Exocet  missile.  Even  though  the 
ship  had  weapons  systems  that  could  have  defeated  the 
Exocet,  its  inability  to  initially  detect  the  missile  at  a 
far  range  rendered  these  defenses  useless.  The  Electronic 
Warfare  (EW)  operators  on  the  SHEFFIELD  had  little  warning 
of  the  Exocet  due  to  self -induced  jamming.  When  the  self- 
jamming  (inadvertent,  of  course)  ceased,  the  Exocet  was 
immediately  detected,  but  it  was  too  late  to  engage.  The 
missile  struck  about  ten  seconds  later.  Although 
technologically  superior,  the  British  did  not  correctly 
manage  the  Radio  Frequency  <RF)  spectrum  and  lost  a  ship. 
The  self-jamming  was  caused  by  equipment  interference  and 
was  either  not  noticed  earlier  or  dismissed  as  unlikely  to 
cause  serious  problems.  This  problem  and  others  like  it  are 
now  being  resolved  by  the  British. 

In   war   at   sea  today,   it  is  necessary  to   provide   an 
adequate  reaction  time.    Reaction  time  is  the  time   between 
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det.ect.ion  of  the  incoming  target  and  weapons  engagement. 
Response  time  is  the  time  between  enemy  weapons  release  and 
impact  (i.e.,  the  time  available  for  detection,  reaction, 
and  engagement) .  In  the  example  cited  above,  that  adequate 
reaction  time  simply  was  not  present.  In  the  case  of  HMS 
SHEFFIELD,  improper  management  of  the  electromagnetic  (EM) 
spectrum  set  up  the  situation  of  inadequate  reaction  time. 
With  the  coming  of  sophisticated  weapon  systems  and 
supersonic  missiles,  the  amount  of  time  available  to  respond 
to  a  threat  has  been  steadily  reduced.  In  World  War  II 
reaction  time  could  be  measured  on  the  order  of  dozens  of 
minutes.  Today  reaction  times  are  on  the  order  of  dozens  of 
seconds.  With  initial  detection  at  the  horizon,  sea- 
skimming  missiles  offer  only  30  seconds  warning  before 
impact.  Today's  Combat  Information  Center  (CIO  needs  to  be 
organized  in  such  a  way  as  to  derive  maximum  efficiency  and 
speed  from  operations  in  order  to  reduce  reaction  time  as 
much  as  possible. 

The  problem  of  reduced  reaction  time  is  not  new  and 
equipments  in  many  areas  has  been  developed  to  meet  this 
need.  The  Navy  Tactical  Data  System  (NTDS)  was  developed  to 
solve  this  problem.  It  reduces  the  amount  of  time  needed  to 
understand  the  tactical  environment  around  the  ship  and  by 
providing  a  more  complete  picture,  aids  sound  decision 
making . 
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With  the  development  and  installation  of  effective 
defensive  weapon  systems  onboard  Navy  ships,  effort  must  now 
be  devoted  to  the  reduction  of  reaction  time.  Effective 
weapons  are  available,  only  the  time  to  employ  them 
correctly  is  needed.  Effective  long  range  sensor  systems 
can  provide  adequate  warning  and  "buy"  time  for  the 
employment  of  the  appropriate  weapon.  Therefore,  it  can  be 
concluded  that  anything  that  "buys  time"  is  of  value. 

But  how  does  one  buy  time?  There  are  two  ways:  (1) 
machines  can  be  built  to  react  more  quickly  or,  (2) 
operators  can  be  trained  to  respond  faster.  Although 
systems  will  help  the  Fleet  sailor  react  quicker,  there  is  a 
limit  as  to  how  fast  he  can  respond.  Working  spaces  need  to 
be  optimized  so  that  the  sailor  can  respond  optimally.  In 
this  context,  efficiency  translates  into  speed  which 
translates  into  reduced  reaction  time.  The  efficient 
arrangement  of  equipment  in  a  working  space  has  not  been 
addressed  by  the  Navy  in  such  a  way  as  to  promote  effective 
and  efficient  spaces.  (For  the  remainder  of  this  effort, 
the  term  "working  space"  or  "work  space"  will  be  used  to 
denote  a  combat  space  where  data  is  searched  for,  collected, 
evaluated,  disseminated,  and/or  acted  upon.)  An  efficient 
and  effectively  laid  out  workspace  will,  intuitively,  buy 
time.  The  barriers  imposed  by  improper  design  and  poor 
layout  can  never  be  totally  compensated  for  by  training. 
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B.   NEED 

During  the  procurement,  cycle,  there  is  a  requirement  to 
perform  human  factors  engineering  on  all  new  equipment  to 
insure  an  adequate  man-machine  interface.  However,  there  is 
inadequate  methodology  for  insuring  that  space  arrangement 
contributes  to  successful  mission  accomplishment.  Space 
arrangement  is,  apparently,  dictated  primarily  by  the  need 
to  fit  new  and  existing  equipment  into  a  space.  This  is  not 
intended  to  belittle  the  efforts  of  those  who  are  charged 
with  designing  the  layouts  and  arrangements  of  combat 
spaces,  but  is  intended  to  address  methods  for  improving  the 
efficiencies  of  layouts. 

Before  the  layouts  of  combat  spaces  can  be  redesigned, 
it  is  first  necessary  to  determine  if  there  is  a  deficiency 
in  the  existing  layout.  A  measure  of  adequacy  of  space 
arrangements  must  be  developed.  At  present  there  is  no  such 
measure  for  combat  systems  layouts. 

The  field  of  human  factors  engineering  has  developed 
techniques  for  assessing  the  adequacy  of  tasks,  subsystems, 
systems,  and  organizations.  However,  due  to  the  dynamic 
nature  of  shipboard  work  space  development  there  appears  to 
have  been  few  human  factors  engineering  techniques 
addressing  the  arrangement  of  systems  as  applied  to  the 
space  as  a  whole  or  the  entire  mission  work  areas.  This  is 
due   to   the  fact  that  new  equipment  with  new  functions   and 
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increased  capabilities  are  constantly  being  introduced   into 
spaces  barely  adequate  for  the  original  equipment. 

As  a  result  of  the  procurement  cycle,  human  factors 
engineering  is  applied  only  on  single  systems  or  consoles. 
It  has  been  recognized  that  training  personnel  to  overcome 
the  human  factors  design  deficiencies  is  not  cost  effective 
in  terms  of  either  time,  money,  or  manpower.  The  current 
requirement  is  for  total  individual  system  analysis  in  the 
areas  of  compatibility,  interoperability,  and  human  factors. 
It  is  mainly  in  the  area  of  integrating  these  systems 
together  in  a  work  space  that  significant  improvement  is 
needed . 

C.   PURPOSE  OF  THESIS 

The  purpose  of  this  thesis  is  to  present  and  discuss  a 
method  of  assessing  the  effectiveness  of  a  work  space 
layout.  In  addition,  this  method  will  provide  the  framework 
for  pinpointing  those  areas  where  redesign  will  be  most  cost 
effective . 
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II.  THE  NATURE  OF  THE  DIFFICULTY 

A.   CURRENT  METHOD 

How  are  apace  layouts  currently  aaaesaed  in  the  United 
Statea  Navy?  Or,  perhaps,  a  better  queation  is:  is  the 
current  method  of  aaaeaaing  apace  arrangements  adequate? 
What  ia  the  current  method? 

There  are  two  methoda  of  improving  a  space  layout.  The 
firat  method  ia  by  fleet  inputs.  There  are  no  formal 
procedures  as  such.  To  initiate  a  design  change, a  request 
for  change  (no  specific  format)  is  submitted  by  the 
individual  (or  ship)  via  his  chain  of  command  to  the 
appropriate  engineering  office  within  the  Naval  Sea  Systems 
Command  (NAVSEASYSCOM) .  An  engineer  studies  the  proposal  to 
see  if  it  has  merit.  If  It  does,  he  forwards  it  within 
NAVSEASYSCOM  and  through  the  appropriate  system  command. 
From  this  point,  if  it  is  acceptable,  it  is  forwarded  to  the 
Office  of  the  Chief  of  Naval  Operations  (OPNAV) .  OPNAV  is 
the  configuration  manager  for  all  platforms  and  makes  the 
final  decision  on  configuration  or  layout  design  changes. 
For  example,  during  the  Board  of  Inspection  and  Survey  (BIS) 
Acceptance  Trials  for  USS  CARL  VINSON  (CVN-70) ,  a  strong 
case  for  redesigning  the  CIC  arrangement  was  made  by  the 
commissioning  crew.  In  particular,  the  cramped  space  of  the 
EW  Module  on  CARL  VINSON  was  addressed  and  a  solution 
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propoaad.  That  solution  apparently  was  forwarded  to 
NAVSEASYSCOM  and  Space  Naval  Warfare  Command  and  then  to 
OPNAV  because  it  has  been  incorporated  in  the  layout  of  the 
EW  Module  on  CVN-71  (See  Figure  1). 

The  other  method  in  improving  space  design  is  a  mock-up 
approach.  At  various  organizations,  mock-ups  are  used  to 
test  the  layout  designs  considered.  These  organizations  are 
under  contract  to  produce  a  specific  kind  of  mock-up.  The 
Naval  Ocean  Systems  Center  (NOSC)  in  San  Diego  does  some 
mock-up  work  under  the  direction  of  NAVSEASYSCOM.  At 
present  they  have  a  carrier  Combat  Information  Center  (CIO 
mock-up.  It  contains  the  Display  and  Decision  portion  which 
includes  the  Surface  Warfare  Module  and  the  Air  Warfare 
Module.  It  does  not  contain  Detection  and  Tracking,  the 
Electronic  Warfare  Module,  or  the  Anti-Submarine  Warfare 
Module. 

B.   DEFICIENCIES 

The  current  method  of  improving  the  layout/arrangement 
design  of  some  of  the  spaces  on  our  surface  ships,  has  four 
major  shortcomings:  (1)  lack  of  user  input,  (2)  lack  of 
human  factors  engineering,  (3)  lack  of  a  learning  curve,  and 
(4)  no  planning  for  growth.  These  deficiencies  and  the 
impact  they  have  on  the  effectiveness  of  the  space  is  now 
discussed . 
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Figure  1.   EW  Module  Layout  (Proposed) 
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1 .  Lack  of  User  Input 

The  current,  method  of  layout  improvement  has  little 
or  no  operator /user  input.  There  is,  perhaps,  some  user 
input  as  in  the  CARL  VINSON  CIC  example  cited  above  but  this 
appears  to  be  the  exception  rather  than  the  rule.  There  is 
no  formal  method  of  submitting  a  design  change  through 
normal  Navy  channels  to  NAVSEA.  This  is  a  very  serious 
deficiency  because  sketchy  or  total  lack  of  fleet  input  is 
counter-productive.  The  design  of  a  space  by  those  who  do 
not  and  will  not  be  using  it  has  a  tendency  to  result  in  a 
far  from  optimal  design.  For  example,  a  radio  or  "bitch 
box"  that  is  frequently  used  is  placed  just  out  of  reach. 
In  the  EW  Module  on  USS  CARL  VINSON,  the  12  MC  (internal 
communications  set)  used  to  communicate  with  the  Tactical 
Action  Officer  (TAO)  and  INTEL (among  others)  is  placed  such 
that  the  EW  Supervisor  and  NTDS  operator  have  to  get  up  out 
of  his  seat  to  talk.  There  is,  in  fairness,  a  hand  mike 
that  can  be  attached  but  this  has  the  undesirable  side 
effect  of  cluttering  the  workstation. 

2.  Lack  of  Human  Factors  Engineering 

The  only  human  factors  engineering  being  employed  is 
basic.  This  method  has  been  characterized  as  "moving  the 
furniture  around".  This  is  done  until  there  is  an 
apparently  workable  solution.  Again  there  is  inadequate 
fleet  or  operator  inputs  to  check  the  "new"  arrangement.  It 
must   be  said  that  those  employing  this  method  are  trying  to 
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find  an  arrangement  that  facilitates  an  efficient  and 
effective  operation.  This  Method  is  the  beat  method 
currently  available  to  accomplish  this  task,  but  still 
something  is  lacking. 

3.  Lack  of  a  Learning  Curve 

However  good  the  results  of  the  mock-up  method  may 
be,  there  is  no  apparent  learning  curve  in  successive 
layouts.  For  example,  the  EW  Module  layout  on  USS  AMERICA 
provided  the  necessary  room  for  the  activities  of  EW  and 
gave  the  impression  of  smooth  efficiency  and  competence. 
However,  on  later  aircraft  carriers  (most  notably,  the 
NIMITZ  class)  the  EW  Module  arrangement  is  a  regression  and 
in  nowise  approaches  the  room  and  layout  effectiveness  found 
on  AMERICA.  If  the  NIMITZ  class  carrier  EW  Module  layout 
was  intended  to  be  an  improvement  over  AMERICA,  it  failed. 

4.  No  plan  for  Growth 

The  current  method  is  deficient  in  terms  of  its 
potential  for  growth.  Few  designs  provide  room  for 
expansion  for  either  new  equipment  or  modifications  to  older 
equipment.  When  new  equipment  is  added,  the  space  for  it 
must  come  from  someplace,  even  if  that  area  has  another 
function.  Simply  adding  new  equipment  does  not  aid  the 
operation  of  the  overall  work  area  and  may  even  be 
counterproductive  in  that  efficiency  may  be  reduced. 

When  lead  units  are  designed  and  built,  they  are 
constructed  with  only  existing  equipment  is  mind  (this  is   a 
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general  rule  and  there  9T&  some  exceptions) .  There  is  some 
small  amount,  of  room  for  expansion  of  capabilities  but  it  is 
thought  that  the  new  equipment  will  replace  older  equipment 
and  take  up  the  same  amount  (or  less)  of  room  and  fit  into 
the  same  space.  This  thought  does  not  take  into  account  new 
missions  for  the  space  with  corresponding  new  equipment,  new 
capabilities,  and  new  space  requirements.  Therefore,  one 
can  readily  see  that  new  equipment  must  be  added  wherever 
there  is  room  for  it.  Sometimes  the  space  where  the  new 
equipment  is  added  is  unsatisfactory  for  the  equipment  and 
its  operation.  By  way  of  example,  USS  CARL  VINSON  is  a 
NIMITZ  class  aircraft  carrier  and  was  built  using 
essentially  the  same  blueprints  as  the  lead  ship.  The  EW 
Module  space  was  not  changed  even  though  the  SLQ-17  and  SSQ- 
82  (MUTE) ,  not  yet  procured  when  NIMITZ  was  designed  and 
built,  were  slated  for  CARL  VINSON  (and  all  carriers, 
eventually) (See  Figure  2) .  Only  one  equipment  rack  was 
removed  to  make  way  for  both  equipments.  The  SLQ-17  console 
fits  into  the  vacated  equipment  rack.  This  still  left  the 
computer  unit  (about  one  and  a  half  racks)  and  MUTE  (which 
was  designed  and  built  wider  than  the  standard  rack)  to  be 
placed  somewhere  within  the  EW  Module.  The  SLQ-17  computer 
rack  was  placed  against  the  bulkhead  in  the  middle  of  the 
apace.  At  most,  three  operators  could  fit  comfortably  into 
the  EW  Module  even  though  General  Quarters  manning  calls  for 
four  operators,  and  MUTE  was  placed  outside  the  EW  Module  in 
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a  passageway  within  CIC  that  was  heavily  traveled.  There 
was  simply  no  room  in  the  EW  Module  for  both  the  EW 
operators  and  MUTE--one  or  the  other  had  to  go. 

In  addition  to  the  problems  cited  above,  to 
accommodate  the  inclusion  of  all  the  equipment  in  the  EW 
Module,  some  severe  space  economies  had  to  be  made.  The 
layout  now  took  on  the  appearance  as  shown  in  Figure  2.  To 
allow  some  passage  of  operators  and  maintenance  people  among 
and  around  the  equipment,  a  "straight  line"  layout  was 
adopted.  This  had  the  sole  advantage  of  allowing  all  the 
equipment  possible  to  be  place  in  the  space.  However,  the 
question  can  logically  be  asked,  "Does  such  an  arrangement 
add  or  detract  from  the  efficiency  and  effectiveness  of 
space  utilization  in  accomplishing  the  mission?"  New 
equipment  added  to  a  space  that  was  not  designed  for  it  may 
cause  integration  problems  due  to  its  intrinsic  nature 
(i.e.,  in  the  equipment  itself),  its  new  location  (e.g.,  the 
SLQ-17  computer  rack),  and  reduced  workspace  (in  our 
example,  several  racks  where  one  used  to  be  to  the  exclusion 
of  another  piece  of  equipment--MUTE) . 

The  remainder  of  this  thesis  will  be  given  over  to 
attempting  to  find  a  workable  solution  to  the  problem  of 
adequately  designing  a  work  space,  in  particular,  an  EW 
Module.  As  indicated  earlier  this  is  an  area  where  the  costs 
are  in  dollars  and  effort,  but  the  payoff  is  in  shorter 
reaction  time  and,  ultimately,  in  ships  and  lives  saved. 
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III.   APPROACH 

A.   IMPROVEMENT  TO  LAYOUTS 

The  solution  to  layout/arrangement  improvement  is 
neither  simple  nor  straightforward.  An  improvement, 
however,  can  be  found  in  a  threefold  approach  to  the 
problem.  These  are:  CD  a  ship  class,  module,  mock-up  at  a 
land  based  laboratory,  (2)  fleet  inputs  added  to  it  on  a 
regular  basis,  and  (3)  a  quantifiable  measure  that  can  be 
used  to  determine  overall  effectiveness  and  pinpoint  problem 
areas. 

Establishing  a  class,  module,  mock-up  at  a  land  based 
laboratory  makes  good  sense.  Here,  the  results  of  several 
mock-ups  can  be  stored  and  compared.  Here,  too,  a  "learning 
curve"  can  be  established.  What  does  not  work  for  one  class 
and  module  may  never  work,  or  it  may  work  for  another  class 
ship  and  another  module.  The  cost  of  mock-ups  can  be  kept 
low.  Mock-ups  of  new  equipment  entering  the  fleet  can  be 
sent  to  just  one  location  and  then  incorporated  into  the 
design  or  redesign.  Mock-ups  of  new  ship  classes  can  easily 
be  done  there. 

NOSC  at  San  Diego  seems  to  be  a  good  place  to  have  this 
mock-up  facility  for  several  reasons.  First,  experts  there 
have  already  done  some  mock-up  work  and  have  a  certain 
amount  of  experience  in  this  area.    Secondly,  they  are  near 
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a  good  source  of  fleet,  inputs  in  San  Diego.  Once  a  mock-up 
was  designed  (or  redesigned) ,  NOSC  could  request  some  fleet 
operators  from  one  of  the  ships  of  that  particular  class  and 
these  operators  could  critique  the  mock-up  and  make 
suggestions  for  improvement.  For  added  realism  and 
additional  inputs,  a  mock  scenario  could  be  played  out  by 
the  operators  on  the  mock-up.  This  has  the  added  possible 
benefit  of  uncovering  any  oversight  by  either  NOSC  or  the 
operators'  critique.  The  two  logical  places  for  the  mock-up 
site  are  Norfolk,  Va .  and  San  Diego,  Ca . 

Fleet  inputs  in  the  design/redesign  of  the  layout 
process  is  of  the  utmost  importance.  The  fleet  operators 
are  the  people  who  have  to  use  the  equipment  and  accomplish 
the  mission  within  the  space.  They,  from  the  benefit  of 
several  years  individual  and  many  years  collective 
experience,  will  be  able  to  note  problems  with  the  mock-up 
that  the  designers  may  have  missed.  Designers  of  single 
equipments  tend  to  think  of  their  equipment  in  isolation 
from  all  others.  Layout  designers  are  often  not  familiar 
with  the  operating  characteristics  of  all  the  equipment. 
Fleet  operators  suffer  from  neither  of  these  deficiencies. 
However,  operators  do  have  a  bias  toward  doing  things  as 
they  are  currently  done  and  may  resist  change.  Nevertheless, 
they  are  still  probably  the  best  ones  to  evaluate  the  mock- 
up. 
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Aa  seemingly  complete  as  the  combination  of  both 
laboratory  mock-up  and  fleet  input  might  be,  there  is  one 
more  area  that  needs  to  be  covered.  This  is  a  quantitative 
assessment  of  the  present  layout  and  mock-up  layouts.  There 
are  several  reasons  for  this.  First,  a  quantitative 
assessment  of  a  present  layout  may  indicate  that  it  does  not 
need  improving  or  that  the  cost  of  improving  the  layout  is 
not  justified  by  the  amount  of  improvement.  Second,  a 
quantitative  assessment  based  in  part  on  questionnaires  to 
fleet  operators  may  awaken  thoughts  of  some  inadequacy  that 
was  not  present  in  the  conscious  memory  but  was  tucked  away 
in  the  recesses  of  the  mind.  Finally,  a  quantitative 
assessment  is  necessary  to  be  able  to  compare  functional 
layouts  one  to  another.  The  final  aspect  of  this  approach  is 
a  way  of  assessing  the  effectiveness  of  the  layout. 

Various  techniques  have  been  developed  that  will  aid  in 
assessing  effectiveness.  However,  these  methods  have  been 
used  on  systems  that  are  dissimilar  to  those  found  on  ships 
and  must  be  modified.  The  method  that  will  be  utilized  is  a 
combination  of  three  different  but  related  techniques:  Task 
Analysis,  Link  Analysis,  and  Operability  Analysis.  Two 
major  studies  have  been  reviewed  to  determine  the  extent  of 
these  analyses  and  how  they  might  be  modified  for  a  layout 
improvement  application.  These  are  Integration  Analysis  and 
Mission   Operability  Assessment  Technique  (MOAT) .    A   brief 
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look  at  each  of  these  will  Indicate  the  salient  portions  of 
each  for  this  application. 

Integration  Analysis  is  the  integration  of  Task 
Analysis,  Operator  Interviews,  and  Link  Analysis  to  evaluate 
a  system's  Functional  Mock-up.  Integration  Analysis  was 
designed  as  a  viable  Test  and  Evaluation  technique  for  the 
earlier  stages  of  Developmental  Test  and  Evaluation  (DTS.E) 
in  order  to  reduce  design  discrepancies  and  minimize 
acquistion  costs  and  time  CRef.  13. 

MOAT,     an    evaluation    methodology,    measures    the 

operability   of  a  system  or  subsystem  in  terms   of   operator 

tasks   performed   during  a  mission.    It  essentially   is   an 

Operability  Analysis. 

In  general,  MOAT  addresses  the  problem  of  how  well  an 
operator  can  use  a  system  or  subsystem  to  perform  tasks 
within  the  mission  context.  Contrasted  to  evaluations 
using  human  engineering  design  criteria  which  present  only 
pass  or  fail  information,  this  technique  provides 
information  on  the  degree  of  system  and/or  subsystem 
success  or  failure. CRef .  2:pp.  3-43 

The    underlying   techniques   of   task    analysis,    scaling 

methodology,   and   multi-attribute  utility  <MAU)  theory  have 

been  integrated  into  one  comprehensive  methodology. 

MOAT  systematically  structures  operator  tasks  in  accordance 
with  mission  needs  and  then  assesses  the  operability  of 
these  tasks  through  conjoint  measurement.  All  assessments 
are  then  integrated  within  rules  established  through  MAU 
theory.  The  output  of  MOAT  is  quantitative  information 
about  the  operability  of  an  entire  system,  such  as  a 
fighter  or  attack  aircraft;  the  operability  of  specific 
subsystems  such  as  radar,  communications,  or  navigation, 
and;  finally,  the  operability  of  each  task  performed  during 
a  mission  phase.  In  short,  MOAT  measures  subsystem  and/or 
system  goal  attainment. CRef.  2:p.  4] 
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Having  briefly  described  both  Int.egrat.ion  Analysis  and 
MOAT,  parts  of  each  were  combined  in  this  thesis  to 
provide  a  technique  that  is  well  suited  to  a  layout 
improvement  application.  From  MOAT  the  techniques  to  assess 
man-system  compatibility  (i.e..  Task  Analysis  and 
Operability  Analysis)  were  used.  This  was  considered  in  a 
larger  context  in  order  to  assess  man  as  a  team  rather  than 
as  an  individual.  Operability  Analysis  consists  of  two 
parts:  multi -attribute  utility  (MAU)  theory  (to  be  discussed 
later),  and  scaling  theory.  The  use  of  questionnaires  and 
Link  Analysis  came  from  Integration  Analysis.  Note  that 
both  Integration  Analysis  and  MOAT  contain  Task  Analysis. 
The  questionnaires  serve  two  purposes.  First,  they  focus 
attention  on  the  problem  areas  of  the  design/arrangement. 
Secondly,  the  completed  questionnaires  support  assessment  of 
the  layout  effectiveness.  Finally,  the  questionnaires  form 
a  link  between  the  various  analyses  and  operator  inputs. 

MOAT  was  designed  to  assess  man-system  compatibility. 
The  original  MOAT  used  a  construct  that  embodied  the  three 
most  important  divisions  of  the  man-system  compatibility-- 
man,  system,  and  mission  and  how  all  three  interact  during 
mission  accomplishment.  The  difference  considered  here  lies 
in  the  evaluation  of  a  team  of  operators  rather  than  just 
one  man  and  the  fact  that  a  group  of  subsystems  arranged  in 
a  particular  manner  is  used  to  determine  the  operability. 
Hence,   the  three  most  important  divisions  of  the  man-system 
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compatibility  are  now  the  team,  system,  and  mission.   As  can 

be   easily  seen,   the  basic  premise  of  MOAT  is  unchanged   in 

that  the  man-system  compatibility  is  still  being   evaluated. 

In   short,   is  the  arrangement  of  equipment  in  a  space  "user 

friendly"?     Note   that   even   though   an   arrangement    of 

equipment    within   a   work   space   is   being    specifically 

addressed,   man-system   compatibility  subsumes  the  equipment 

arrangement  within  the  work  space.    The  basic  contention  is 

that   the   best  operators  and  best  subsystems   in   a   poorly 

designed   space   may  be  less  effective  than   operators   with 

less   ability   and  a  less  capable  system  that  is  in  a   space 

that  is  well  designed  for  mission  accomplishment. 

MOAT   uses  the  term  operability  to  reflect  how  the   man, 

system,  and  mission  interact  during  mission  accomplishment. 

By  definition,  operability  reflects  (1)  the  amount  of 
effort  required  by  the  operator  in  task  accomplishment,  (2) 
the  degree  of  subsystem  technical  effectiveness  in  aiding 
the  operator  in  task  accomplishment,  and  (3)  how  important 
the  task  is  for  mission  success. CRef.  2:p.  193 

This  can  be  redefined  slightly  to  indicate  (1)  the  amount  of 

effort    required    by    the   operators    (team)    in    task 

accomplishment   (task   difficulty) ,    (2)   the   degree   that 

equipment   arrangement   aids   the  operators  (team)   in   task 

accomplishment   (arrangement   effectiveness),   and   (3)   how 

important    the    task    is   for    mission    success    (task 

criticality) . 
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B.   TASK  ANALYSIS 

The  purpose  for  Task  Analysis  is  to  determine  those 
tasks  and  subtasks  needed  to  successfully  perform  the 
mission  of  the  Module.  Without  specifying  the  tasks 
performed  within  the  EW  Module,  it  would  be  difficult,  if 
not  impossible,  to  determine  layout  effectiveness, 
task/subtask  identification  forms  the  basis  of  both  the  Link 
Analysis  that  is  discussed  later  and  the  MOAT  technique  of 
Operability  Analysis.  Each  task  and  subtask  that  operators 
perform  will  be  examined  and  fit  into  the  larger  picture  of 
module  mission.  The  effort  within  the  EW  Module  can  be  shown 
to  be  divided  hierarchically:  Module  mission,  operator 
tasks,  and  operator  subtasks.  This  hierarchy  is  divided  in 
the  following  manner:  the  aggregate  of  the  subtasks 
comprises  the  individual  task  and  the  aggregate  of  the 
individual  tasks  comprises  the  mission. 

The  mission  of  the  EW  Module  is  to  conduct  Electronic 
Warfare  which  includes  Electronic  Warfare  Support  Measures 
<ESM)  and  Electronic  Warfare  Counter-Measures  (ECM).  This 
entails  attempting  to  deny  any  potential  enemy  the 
exploitation  of  the  electromagnetic  spectrum  while 
preserving  it  for  our  own  use.  The  EW  Supervisor  (EWS)  is 
responsible  for  providing  timely  evaluated  EW  information, 
EW  data,  and  EW  control  (to  the  rest  of  the  battle  group 
when  so  designated  as  EW  Control  Ship) .  This  is  accomplished 
by  three  operators  and  three  work  stations   (WLR-1,   SLQ-17, 
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and  NTDS) .  The  tasks  and  subtasks  that  are  performed  within 
the  workspace  are  listed  in  Table  1.  While  the  module 
mission  delineates  the  overall  responsibility  for  the 
module,  the  operator  tasks  are  the  first  major  subdivision. 
These  are  the  tasks  that  each  operator  must  accomplish  at 
his  workstation  in  order  to  contribute  to  mission 
accomplishment.  The  operator  subtask  is  a  further  division 
of  the  operator  tasks.  These  aggregate  together  for 
ESM/ECM.  These  are  listed  in  Table  1  and  were  drawn  from 
various  sources  and  confirmed  by  the  EW  Module  personnel . 
Each  workstation  and,  therefore,  each  operator  has  a  piece 
of  the  "puzzle"  and  only  by  putting  them  together  can  any 
sense  be  made  out  of  the  parts.  In  this  case,  as  so  many 
others,  the  whole  will  be  greater  than  the  sum  of  the  parts. 
Note  in  Table  1  that  there  are  actually  five  positions 
listed:  EW  Supervisor,  WLR-1  operator,  SLQ-17  operator,  NTDS 
operator,  and  Status  Board  keeper.  During  normal  steaming 
conditions,  one  of  the  three  position  operators  is  also  the 
EW  Supervisor  and,  therefore,  he  has  a  dual  role  to  play. 
Additionally,  there  is  no  Status  Board  keeper  during  normal 
steaming  watches.  During  Condition  One,  General  Quarters, 
the  Module  is  manned  with  five  people.  Therefore,  the  Task 
Analysis  considered  the  more  complicated  situation  of 
General  Quarters. 
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TABLE  1.   OPERATOR  TASKS  AND  SUBTASKS 

EW  Supervisor 

Task:  1 . 1  Direct  ESM  search 
Subtaska 

1.1.1  Assign  search  parameters  to  SLQ-17 

1.1.2  Assign  search  parameters  to  WLR-1 

1.1.3  Assign  ESM  sensor  report  responsibilities   --  own  ship 

1.1.4  Assign  ESM  sensor  report  responsibilities   --  force 

1.1.5  Initiate  manual  ID  request  -  ship 

1.1.6  Initiate  manual  ID  request  -  force 

1.1.7  Monitor  automatic  correlations/associations,  (SLQ-17) 

Task;  1.2  Report/Disseminate  EW  Information 
Subtaska 

1.2.1  Report  evaluated  EW  information 

1.2.2  Provide  EW  recommendations 

1.2.3  Update  status  board  near  NTDS  console 

1.2.4  Brief /debrief  embarked  Airwings 

1.2.5  Navigation  by  passive  EW 

Task:  1.3  Counter  Hostile  Environment 

Subtaska 

1.3.1  Promulgation  of  ECM  employment  criteria 

NTDS  Operator 

Task:  2.1  Collect  and  enter  EW  data  into  NTDS 
Subtaska 

2.1.1  Enter  manual  ID  information  into  NTDS 

2.1.2  Enter  manual  ESM/NTDS  track  associations 

2.1.3  Perform  triangulation  of  ESM  bearing  lines 

2.1.4  Enter  EW  fixes 

2.1.5  Advise  operators  of  bearing  resolution 

2.1.6  Evaluate  externally  reported  ESM  bearings 

Task :  2.2  Report/Disseminate  EW  Information 
Subtask 

2.2.1  Report  evaluated  EW  information 

2.2.2  Update  status  board  near  console 

SLQ-17  Operator 

Task:  3.1  Conduct  ESM  Search 
Subtask 

3.1.1  Monitor  automatic  correlations/associations 

3.1.2  Establish  operating  modes  of  SLQ-17  (ESM) 

3.1.3  Enter  detection  and  response  parameters  (ESM/ECM) 

3.1.4  Monitor  environment  on  NTDS  console 

3.1.5  Evaluate  displayed  data 
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TABLE  1.   OPERATOR  TASKS  AND  SUBTASKS  (continued) 

Task.:  3.2  Report/Disseminate  EW  Information 
Subtaak 

3.2.1  Report  evaluated  EW  information 

3.2.2  Provide  ESM/ECM  data  to  NTDS 

3.2.3  Monitor  entry  of  EW  data  into  NTDS 

3.2.4  Update  status  board 

Task;  3.3  Counter  Hostile  Environment 
Subtask 

3.3.1  Engage  targets  with  ECM 

3.3.2  Establish  ECM  operating  Modes 

3.3.3  Assist  in  promulgation  of  ECM  employment  criteria 

WLR-1  Operator 

Task:  4.1  Conduct  ESM  Search 
Subtasks 

4.1.1  Search  assigned  bands 

4.1.2  Analyze  intercepted  signals 

4.1.3  Check  intercepts  for  images/harmonics 

4.1.4  Accurately  DF  intercepted  signals 

4.1.5  Assist  in  evaluating  ECM 

Task :  4.2  Report/Disseminate  EW  Information 
Subtaska 

4.2.1  Provide  ESM  data  to  NTDS 

4.2.2  Report  evaluated  EW  information 

4.2.3  Update  status  board  near  position 

4.2.4  Log  all  intercepts 

Task:  4.3  EMCON 
Subtasks 

4.3.1  Monitor  EMCON 

4.3.2  Report  violations  of  EMCON 

4.3.3  Log  violations  of  EMCON 

4.3.4  Monitor  MUTE 

EW  Status  Board 

Task:  5.1  Maintain  Status  Boards 
Subtasks 

5.1.1  Communicate  with  operators 

5.1.2  Advise  operators  of  any  information  received 

5.1.3  Update  status  boards 
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C.   LINK  ANALYSIS 

Link  Analysis  is  a  technique  that  will  provide  the 
information  needed  to  produce  an  acceptable  arrangement  of 
men  and  machines  in  a  system  CRef .  3: p.  2043 .  The  idea  is 
that  the  "best  arrangement"  can  be  found  only  by  optimizing 
different  types  of  links  that  are  important  in  the 
particular  system  being  designed.  By  way  of  definition,  a 
link  is  a  connection  between  (a)  an  operator  and  a  machine 
or  <b>  two  operators  CRef.  3:p.  2043.  These  links  may  be 
visual  (such  as  an  instrument  scan) ,  functional  or  manual 
(hand  to  control),  or  verbal  (communications). 
Inefficiencies  are  present  when  links  are  comparatively 
long,  crossing  one  another,  blocked,  or  outside  optimal 
visual  or  reach  envelopes.  The  links  are  produced  from  the 
task  analysis  and  illustrate  all  the  operator-required 
functional,  visual,  and  communication  tasks.  Link  Analysis 
can  be  applied  to  all  scenarios  involved  during  all 
operational  and  emergency  conditions  CRef.  3:p.  205  and 
Ref.43.  Link  Analyses  are  normally  of  two  types:  panel 
layout  and  tactical  compartment  or  multiple  operator  work 
area.  With  the  development  and  procurement  of  individual 
subsystems  (i.e.,  WLR-1,  SLQ-17,  etc.)  a  certain  amount  of 
panel  layout  link  analysis  has  been  done.  However,  little 
if  any  has  been  done  on  the  combination  of  systems  arranged 
in  a  workspace  (in  this  example,  the  EW  Module).   Hence,  the 
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Link   Analysis  will  necessarily  be  of  the  latter  type  (i.e., 
multiple  operator  work  area) . 

The  multiple  operator  work  area  type  of  link  analysis  is 
dependent  on  the  correlation  matrix.  Beginning  with  the 
correlation  matrix  and  an  area  layout,  all  interactions 
(links)  required  to  perform  a  particular  functional  task  are 
examined  in  terms  of  the  frequency  with  which  they  occur  and 
their  criticality.  If  the  criticality  is  assigned  a 
numerical  value,  it  may  be  multiplied  by  the  frequency  in 
order  to  obtain  a  weighted  link  value.  The  work  area  is 
overlaid  with  the  weighted  links  permitting  a  picture  of  all 
the  interactions  taking  place  within  the  system  being 
analyzed.  The  system  design  can  then  be  modified  to  shorten 
the  distance  between  the  workstations  that  are  connected  by 
the   weighted   links  CRef.  53. 

Figure  3  contains  the  correlation  matrix  for  the  EW 
Module  in  CARL  VINSON.  A  correlation  matrix  is  a  figure 
that  provides  an  indication  of  the  links  between  two 
operators,  positions,  or  between  an  operator  and  a  position. 
Usually  a  criticality  associated  with  the  particular  links 
is  included  in  the  matrix.  In  Figure  3,  only  the  links  of 
interest  are  listed.  The  figure  is  read  by  selecting  the 
two  entries  for  which  links  are  desired  and  reading 
diagonally  down  from  the  top  one  and  diagonally  up  from  the 
bottom  one  until  the  intersecting  diamond  is  reached.  The 
diamond   contains  both  the  particular  links  between  the   two 
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entries  and  the  criticality  of  those  links.  The  notation 
along  the  right  side  of  the  figure  indicates  what  links  are 
between  each  component  of  the  module.  A  separate  notation 
provides  an  indication  of  the  criticality  of  that  particular 
link:  most  critical,  critical,  and  least  critical.  The  most 
critical  link  is  one  that  is  absolutely  essential  to 
accomplish  the  mission.  A  critical  link  is  one  that 
prevents  severe  degradation  in  the  accomplishment  of  the 
mission  and  the  least  critical  link  has  a  small  impact.  It 
remains  only  to  multiply  the  links  by  a  frequency  of 
operation  to  obtain  a  Weighted  Link  Value.  The  Weighted 
Link  Value  will  be  obtained  from  the  results  of  the  Link 
Analysis  Questionnaire. 

The  type  of  links  are  communication,  visual,  and  manual. 
The  communication  links  are  further  subdivided  into  internal 
and  external.  The  internal  communication  links  are  the 
voice  interaction  between  operators  while  the  external 
communication  links  are  those  voice  and/or  electronic  links 
with  other  modules,  persons,  and/or  platforms.  The  visual 
links  can  also  be  divided  into  internal  and  external.  The 
internal  are  concerned  solely  with  those  links  between  the 
operator  and  his  equipment  or  console.  The  external  are 
those  between  the  operator  and  other  consoles  and/or 
equipment.  Similarly,  the  manual  links,  internal  and 
external,  are  between  the  operator  and  his  equipment  or 
console    and   the   operator   and   other   equipment    and/or 
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consoles,  respectively.  Since  we  have  already  considered 
that,  link  analysis  may  have  been  done  on  individual  systems 
and  our  concern  is  for  the  multiple  operator  work  area,  we 
will  not  be  concerned  with  the  internal  visual  and  internal 
manual  links.  This  leaves  just  four  links  analyses  to  be 
done;  the  internal  and  external  communications,  the  external 
visual,  and  external  manual  links. 

When  the  internal  communication  links  are  considered,  it 
is  noted  that  there  are  links  between  the  EW  Supervisor  and 
both  the  WLR-1  and  the  SLQ-17  operators  to  promulgate 
orders.  Next,  there  are  links  back  to  the  EW  Supervisor 
when  one  or  the  other  have  found  either  the  signal  of 
interest  or  something  else  that  may  be  of  interest.  There 
are  also  links  between  the  WLR-1  operator  and  the  SLQ-17 
operator.  This  last  may  be  queries  for  information  about 
their  particular  equipment  set  up  or  the  passing  of 
information  to  directing  the  other's  search.  The  links 
between  all  three  may  be  in  the  form  of  equipment  status  or 
failure.  The  internal  communication  links  are  shown  in 
Figure  4. 

The  external  communications  involve  primarily  the  EW 
Supervisor  with  any  of  the  following:  TAO,  INTEL  (the 
Intelligence  center),  SURFACE,  AIR,  Trackers,  ASW  (Anti- 
submarine Warfare)  Module,  SSES  (Ships  Signal  Exploitation 
Space),  FLAG,  FWC  (Force  Weapons  Coordinator),  or  SWC 
(Ship's  Weapons  Coordinator). 

37 


u 

Q) 

■M 

/ 

•H 

T3 

U 

ti 

5 

fd 

<u 

0 

Q< 

GO                           r 

> 

Ph 

to 

m 

■p 

GO 


"' I 


2, 


11'rHl       i 

i    i  !       [ 

Jj*!  I 

£ 
O 

a 


-a 
u 
<n 
o 

CQ 


ApmsSpsspj 


CO 

c 

•H 
J 

CO 

c 
o 

•H 
■P 
rd 

o 

•H 

C 

E 
E 

O 

o 


03 
C 

0) 


0) 
■H 


(riTi^; 


•nci 


en 
3 
fJ 
03 
-M 
CO 


38 


The  external  viaual  links  involve  all  of  the  operators 
viewing  the  status  boards  and  each  other's  workstation. 
These  external  visual  links  are  important  for  the  additional 
information  they  provide  to  the  operator.  The  visual 
reference  to  the  other  workstation  builds  an  internal 
working  model  of  the  environment  within  the  operator's  mind 
thus  enabling  him  to  more  quickly  fit  new  data  into  the 
tactical  picture  and  anticipate  subsequent  events.  Without 
this  interaction,  effective  Electronic  Warfare  control  can 
not  be  attained. 

Figures  5  and  6  show  the  external  visual  links.  Figure  5 
shows  visual  links  between  operators  and  Figure  6  indicates 
visual  links  between  the  operators  and  the  status  boards 
(there  are  only  two  status  boards  that  can  be  clearly  seen). 

The  final  link  that  will  be  considered  is  the  external 
manual  link.  Although  there  is  no  requirement  for  an 
operator  to  control  more  than  one  workstation,  there  are 
some  external  manual  links  that  must  be  addressed.  For 
example,  all  those  equipments  that  are  part  of  the  EW  Module 
for  which  there  is  no  manning  authorized  will  fall  under 
this  category.  The  AN/SSQ-82  MUTE  is  a  prime  example.  MUTE 
is  a  monitor  device  that  requires  no  manning  and  only 
cursory  glances  to  ensure  that  it  is  functioning  correctly. 
When  adjustment  is  needed  due  to  Force  or  ship  EMCON 
changes,  an  external  manual  link  for  one  of  the  operators 
takes  place.   Another  example  is  the  SLQ-17  UYK-20  computer. 
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This  equipment,  ia  not  adjacent  to  the  SLQ-17  and  one  of  the 
operators  (normally  the  SLQ-17  operator)  may  need  to  reload, 
reboot,  or  reconfigure  the  system  in  the  event  of  a  casualty 
or  normal  operations.  Figure  7  shows  the  external  manual 
links. 

D.   OPERABILITY  ANALYSIS 

In  the  introduction  to  this  section,  it  was  stated  that 
Operability  Analysis  was  comprised  of  MAU  and  scaling 
theory.  MAU  is  a  Bayesian-oriented  decision-making 
paradigm.  There  are  three  major  aspects  of  the  MAU  model 
which  are  particularly  important  to  this  application.  First, 
the  basic  structure  principle  in  MAU  is  hierarchical 
decomposition.  The  mission  is  broken  down  into  hierarchical 
grouping  of  tasks  and  subtasks.  The  model  provides  the 
structure  and  rules  necessary  to  investigate  and  integrate 
the  interrelationships  of  all  these  tasks  and  subtasks. 
Second,  the  definition  of  utility  used  in  the  MAU  model 
allows  for  the  optimum  evaluation  of  alternatives  which  is 
dependent  upon  the  selection  of  a  single  criterion.  This 
means  that  multidimensional  outcomes  must  be  transformed 
into  a  single  figure  of  merit  such  as  utility,  system  worth, 
system  effectiveness,  or,  as  in  this  application, 
operability.  Third,  a  scaling  of  the  selected  criterion. 
The  scaling  methodology  used  in  this  application,  as  in 
MOAT,  was  conjoint  measurement. 
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Recall  that,  operabillty  can  be  viewed  as  a  function  of 
task  critical ity,  operator  workload,  and  space 
ef f ectiveneaa.  Therefore,  when  considering  each  task  from  a 
operability  standpoint,  each  task  that  is  performed  has  some 
combination  of  these  three  dimensions.  There  is  a 
difficulty  in  assessing  the  degree  of  each  attribute  and 
combining  them  into  a  meaningful  measure  of  operability. 
Since  this  can  not  be  assessed  directly  by  objective 
methods,  the  scaling  methodology  of  conjoint  measurement  was 
devised  to  assess  space  operability  subjectively.  The 
problem  of  scaling  tasks  in  dimensions  of  criticality, 
frequency,  and  system  effectiveness  has  been  successfully 
solved  by  using  objectively  anchored  rating  scales 
CRef.  2:p.  20].  Therefore,  a  similar  rating  scale  procedure 
seemed  suitable  in  this  instance. 

The  major  difficulties  involved  with  this  approach  are 
those  of  measuring  the  degree  of  operator  effort  (or  watch 
section  effort)  and  the  layout  effectiveness.  This  was  to 
be  expected,  however,  since  not  only  were  these  different 
from  any  known  previous  study  but  also  involved 
interactions  on  a  higher  scale  than  that  experienced  before. 
There  is  a  substantial  correlation  in  rating  of  task 
difficulty  and  subsystem  effectiveness.  The  attempt  to 
solve  the  rating  scale  problem  is  accomplished  by  dividing 
it  into  two  separate  ratings.  On  the  F/A-18  program  it  was 
desirable   to  have  two  ratings;   one  with  respect   to   pilot 
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workload  <PW)  and  one  with  respect  to  the  technical 
effectiveness  (TE)  of  the  subsystem  CRef .  2:p.  203 .  A 
similar  approach  is  attempted  here  with  operator  workload 
(0W)  (it  is  assumed  that  this  can  be  directly  translated 
into  watch  section  workload)  and  space  effectiveness  (SE) . 
This  application  of  Operability  Analysis  is  concerned  with 
assessing  the  task  critical ity,  the  operator  workload,  and 
the  space  effectiveness  of  a  module  layout.  All  are  values 
on  an  ordinal  scale.  Two  of  these,  space  effectiveness  and 
operator  workload,  need  to  be  upgraded,  through  some 
methodology,  to  an  interval  scale  in  order  to  aggregate  them 
over  all  tasks  to  achieve  an  overall  measure  of  Module 
operability.  To  this  end,  conjoint  measurement  and  its 
associated  scaling  procedures  seemed  suitable  for  a 
transformation  to  the  desired  unidimensional  interval  level 
scale.  It  is  here  that  the  delta  method  was  employed.  What 
conjoint  measurement  and  the  delta  method  do  is  allow 
separate  rating  of  0W  and  SE  (despite  their  mutual 
dependency)  to  be  obtained  and  be  combined  in  such  a  manner 
that  a  one-dimensional  scale,  having  interval  properties,  is 
created.  This  scale  might  just  as  well  be  called  the 
combined  OW-SE  scale.  Rating  on  this  scale  can  then  be 
plugged  into  the  MAU  model.  An  assessment  of  the 
effectiveness  of  the  EW  Module  layout  with  respect  to  a 
certain   subtask  can  be   determined.    Aggregated   together. 
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these   will    provide    an   indication    of    the    overall 
effectiveness  of  the  layout  upon  mission  accomplishment. 

E.   QUESTIONNAIRES 

Link  Analysis,  Task  Analysis,  and  Operability  Analysis 
will  be  completed  by  a  series  of  two  questionnaires.  The 
first  questionnaire  (the  Link  Analysis  Questionnaire)  was 
targetted  at  Link  Analysis  and  provided  the  frequency 
component  for  a  completed  Link  Analysis.  The  second 
questionnaire  (the  Operator  Subtask  Questionnaire)  confirmed 
the  Task  Analysis  that  went  into  building  it  and  also 
provided  the  raw  data  needed  to  perform  the  Operability 
Analysis.  From  the  Operability  Analysis  came  the  assessment 
of  the  Total  Module  Operability  (TMO) . 

The  questionnaires,  as  shown  in  Appendix  A,  were 
designed  to  do  two  things.  First,  the  Link  Analysis 
Questionnaire  gave  a  general  idea  of  the  type  and  degree  of 
deficiencies  in  the  space  in  terms  of  link  deficiencies. 
Second,  it  focused  the  operators'  attention  on  the 
arrangement  of  the  work  space  and  any  deficiencies  that  were 
there.  It  was  hoped  that  the  questions  brought  into  sharp 
relief  the  difficulties  for  which  the  operator  unconsciously 
compensates  during  the  mission.  By  highlighting  these 
deficiencies  through  the  Link  Analysis  series  of  questions, 
the  detailing  of  them  in  the  Operator  Subtask 
Questionnaires,   hopefully,   provided  good  human  engineering 
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data  with  which  to  evaluate  the  Module  Operability  of  the  EW 
Module. 

The  Link  Analysis  Questionnaire  was  designed  to  determine 
the  frequency  of  the  various  links  in  the  EW  Module. 
Combined  with  the  correlation  matrix  that  indicates 
crlticality,  this  questionnaire  determined  the  weights  of 
the  various  links.  This  weighting  indicated  the  most 
heavily  used  links.  This,  in  turn,  can  focus  attention  on 
deficiencies  in  these  links.  A  possible  example  of  this 
might  well  be  the  abnormally  long  internal  communication 
link  between  the  NTDS  console  and  the  WLR-1  position. 

The  Operator  Subtask  Questionnaire  contained  a  section 
requiring  an  assessment  for  each  of  the  forty-nine  subtasks 
delineated  in  Table  1.  The  subtasks  were  drawn  from  a 
variety  of  sources  (including  USS  CARL  VINSON  Combat 
Direction  Center  doctrine)  and  verified  by  the  EW  operators 
prior  to  the  administration  of  the  questionnaire.  This 
assessment  was  the  culmination  of  the  Task  Analysis  and  the 
beginning  of  the  Operability  Analysis.  In  the  Subtask 
Assessment,  each  subtask  on  the  questionnaire  was  evaluated 
by  the  EW  operators  with  regards  to  Operator  Workload,  Space 
Effectiveness,  and  Criticality  of  the  Subtask  to  the  overall 
mission  accomplishment. 

The  Operator  Subtask  Questionnaire  also  contained  the 
Ranking  Matrix,  where  combinations  of  the  various  degrees  of 
Operator   Workload   <0W)  and  Space  Effectiveness   (SE)   were 
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TABLE  2.  RANKING  MATRIX 
In  the  following  matrix,  the  blocks  are  ranked  from 
best  to  worst  <1  to  16).  The  lowest  numbered  block  is  the 
intersection  of  the  best  of  the  rows  and  columns.  The  number 
two  block  is  next  best,  and  so  on.  Note  the  arrows  and  the 
phrases  associated  with  them.  Design  means  the  design  of 
the  layout  or  arrangement. 
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ranked.  Thia  was  of  the  utmost.  importance  since  the 
conjoint  measurement  of  the  subtask  assessment  and  Module 
operability  depended  upon  it.  Table  2  contains  a  blank 
Ranking  Matrix.  The  EW  personnel  were  asked  to  rank  the 
intersections  from  best  to  worst  for  the  "typical"  subtask. 
It  was  assumed  that  the  rank  order  for  the  matrix  would  vary 
little  from  subtask  to  subtask.  Helms  found  this  to  be  true 
CRef.  2: p.  34].  This  may  have  been  the  most  difficult  part 
of  the  questionnaire  and  the  EW  operators  were  forced  to 
draw  upon  all  their  knowledge  and  previous  experience  in 
order  to  produce  a  rank  order  that  was  meaningful  and 
replicable.  This  matrix,  the  intersection  of  two  ordinal 
scales  (OW  and  SE) ,  is  part  of  conjoint  measurement.  The 
Ranking  Matrix  was  expanded  and  an  interval  scale 
constructed  via  a  linear  expansion  know  as  the  delta  method. 
This  resulted  in  an  interval  scaling  from  0  to  100  and  was 
used  to  evaluate  the  total  Module  operability.  The  delta 
method  of  converting  two  of  these  ordinal  scales  to  an 
interval  scale  is  described  in  Appendix  C. 

Using  this  interval  scale,  the  intersection  of  any 
particular  set  of  Operator  Workload  and  Space  Effectiveness 
values  on  the  returned  Operator  Subtask  Questionnaire  gives 
a  predetermined  Operability  Value  between  0  and  100.  An 
Operator  Workload  value  between  one  and  four  inclusive 
served  to  identify  a  column  while  a  Space  Effectiveness 
value   identified  a  row.    The  intersection  of  the  row   with 
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the  column  indicated  the  assessment  of  that  particular 
subtask  by  an  operator.  For  every  subtask,  this  Operability 
Value  was  obtained  for  each  rater  and  the  mean  and  standard 
deviation  were  calculated.  This  mean  value  represented  the 
Operability  Value  for  that  subtask. 

The  remaining  ordinal  scale  is  that  of  the  Criticality. 
There  was  no  attempt  to  convert  this  to  an  interval  scale. 
Although  operators'  skills  might  vary,  causing  significant 
deviations  in  the  ratings  from  rater  to  rater,  there  should 
be  only  one  standard  for  the  criticality  of  a  subtask  as  it 
relates  to  mission  accomplishment.  This  single  measure  of 
criticality  was  taken  to  be  the  mean  of  the  criticality 
ratings.  The  Operability  Value  was  multiplied  by  the 
criticality  resulting  in  a  Weighted  Operability  Value.  A 
Weighted  Deficit  Value  was  computed  as  (100  -  Operability 
Value)  multiplied  by  the  Criticality  of  the  subtask. 
Whereas  the  Weighted  Operability  Value  will  give  an 
indication  of  the  "goodness"  of  the  layout  for  a  particular 
subtask,  the  Weighted  Deficit  Value  gives  an  indication  of 
how  much  improvement  is  required  to  optimize  Module 
Operability  for  a  particular  subtask. 

The  Link  Analysis  Questionnaire  was  given  approximately 
one  week  before  the  Operator  Subtask  Questionnaire.  It  was 
hoped  that  the  brief  exposure  to  the  first  questionnaire 
increased  the  accuracy  of  the  second. 
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IV.   RESULTS 

To  teat,  thia  application,  a  auitable  platform  was 
required.  The  Electronic  Warfare  Module  on  an  U.S.  Navy 
aircraft  carrier  waa  selected.  The  particular  ship,  USS 
CARL  VINSON  (CVN-70) ,  was  chosen  for  three  reasons: 
availability,  accessibility,  and  familiarity.  CARL  VINSON 
had  just  returned  from  a  seven  month  cruise  and  was  in  a 
stand-down  period  and,  so,  available.  The  ship's  homeport, 
Alameda,  Ca . ,  was  readily  accessible  for  the  test.  Finally, 
the  ship's  layout  was  familiar  enough  to  the  test  director 
to  allow  a  minimum  amount  of  time  to  be  spent  on  the  ship 
and,  therefore,  lessen  the  impact  upon  the  ship's  daily  work 
and  schedule. 

There  were  limitations  to  the  scope  of  testing.  First, 
the  test  was  not  done  at  sea  which  produced  two  limitations. 
In  regards  to  Link  Analysis,  operator  usage  of  the  various 
links  and  the  associated  frequencies  could  not  be  monitored. 
This  was  considered  to  be  a  major  limitation  in  regards  to 
only  the  Link  Analysis  portion  of  the  test.  The 
compensation  for  this  was  the  Link  Analysis  Questionnaire 
concerning  the  frequency  of  link  usage.  A  minor  limitation 
concerned  the  inability  to  observe  the  actual  Subtasks  and 
ascertain   the  criticalities  under  actual   conditions.   This 
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was  compensated  by  the  Operator  Subtask  Questionnaire,  which 
was  considered  adequate. 

A  further  limitation  was  the  small  number  of  valid 
responses  for  the  questionnaires.  There  were  three  valid 
responses  for  the  Link  Analysis  Questionnaire,  five  for  the 
Rank  Ordering  portion  of  the  Operator  Subtask  Questionnaire, 
and  from  five  to  seven  for  the  Subtask  Assessment  portion  of 
the  Operator  Subtask  Questionnaire.  While  these  numbers  are 
small  from  a  statistical  point  of  view,  they  can  not  be 
discounted.  The  limited  sample  size  should  be  an  inducement 
for  further  testing.  Furthermore,  the  sample  size  for  any 
aircraft  carrier  will  never  be  much  greater  than  about 
twelve  due  to  manning  levels.  The  sample  size  was  seven  due 
to  leave  and  various  schools  but  included  the  personnel  with 
the  most  experience.  In  may  be  argued  that  not  testing 
other  platforms  is  a  limitation.  However,  since  no  two  EW 
Modules  on  U.S.  aircraft  carriers  are  alike,  the  lack  of 
multiple  testing  is  a  moot  question. 

The  test  was  conducted  in  the  EW  Module  of  USS  CARL 
VINSON  (CVN-70) .  The  Module  was  used  so  that  the  personnel 
could  refresh  their  memory  with  regards  to  the  layout  as 
they  evaluated  the  subtasks  in  relation  to  the  layout. 

The  guidance  given  to  the  EW  personnel  before  and  during 
the  test  stressed  that  they  could  ask  any  question  they 
wished   of   anyone   they  wished.    They  were   encouraged   to 
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confer   with  each  other  about  the   workload,   effectiveness, 
and  criticality. 

A.   LINK  ANALYSIS 

The  results  of  the  Link  Analysis  were  taken  from  the 
Link  Analysis  figures  and  from  the  Link  Analysis 
Questionnaire.  The  questionnaire  was  produced  from  the  Link 
Analysis  figures  and  the  Task  Analysis  in  order  to  determine 
the  frequency  that  these  links  were  used.  The  EW  operators 
on  USS  CARL  VINSON  were  asked  to  estimate  how  many  times 
during  a  standard  eight  <8)  hour  watch  they  utilized  the 
links.  The  Link  Analysis  Questionnaire  is  listed  in 
Appendix  A  and  the  results  of  the  Link  Analysis  is  shown  in 
Table  3. 

1 .  Link  Analysis  Figures 

The  most  critical  links  were  assessed  to  be  the 
communication  links  between  operators  and  the  visual  links 
between  positions.  The  criticality  of  the  links  were  chosen 
to  reflect  mission  accomplishment  and  the  frequency  of  usage 
confirmed  the  criticality.  There  were  four  links  considered 
in  the  Link  Analysis:  internal  communication,  external 
communication,  external  visual,  and  external  manual.  Of 
these  four,  the  two  most  important  links  are  the  internal 
communications  and  external  visual.  This  is  because  the 
external  communication  will  generally  involve  only  one 
operator   (the   EW  Supervisor/NTDS  operator)   and   there   is 
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Position  and  Tasks 


TABLE  3.  LINK  ANALYSIS  BY  POSITION 


Frequency  Link  Critical lty 


Weignted 
Link  Value 


HUM  Operator: 

i.  Talk/comunicate  with  the  SLQ-17  operator? 

2.  View  the  presentation  on  the  SLQ-17  console? 

3.  Talk/comunicate  with  the  NTDS  operator? 

4.  View  the  presentation  on  the  NTDS  console? 

5.  View  the  NTDS  Status  Board  (SB) ? 
S.  View  the  MUM  Status  Board? 

7.  Update  the  WUM  Status  Board? 

8.  Check  (visually)  the  SLQ-17  computer? 

9.  Reboot,  reset,  or  work  with  the  SLQ-17  computer? 
13.  Check  «JTE? 

11.  Change  any  settings  on  MUTE? 


16.667 

IC 

3 

77.333 

EV 

3 

44.889 

IC 

3 

73.333 

EV 

3 

33.333 

EV 

1 

33.333 

EV 

a 

1.667 

EM 

2 

31.088 

EV 

1 

1.667 

EM 

1 

3.080 

EV 

2 

0.667 

EM 

2 

50.00 

231.99 

132.80 

219.99 

33.33 

66.67 

3.34 

31.80 

1.67 

6.00 

1.34 


SLQ-17  Operator: 


6. 
7. 

8. 
9. 

10, 


Talk/communicate  with  the  NTDS  operator? 

View  the  presentation  on  the  NTDS  console? 

Talk/communicate  with  the  HUM  operator? 

View  the  presentation  on  the  WLR-1  console? 

View  the  WLR-1  Status  Board? 

View  the  MTDS  Status  Board? 

Uodate  the  NTDS  Status  Board? 

Check  (visually)  the  SLQ-17  computer? 

Reboot,  reset,  or  worn  with  the  SLQ-17  computer? 

Check  MUTE? 

Change  any  settings  on  MUTE? 


54.000 

IC 

3 

162.88 

73.333 

EV 

3 

219.99 

71.667 

IC 

3 

215. 88 

48.333 

EV 

3 

144.99 

23.080 

EV 

& 

59.80 

31.333 

EV 

3 

93.99 

9.000 

EM 

1 

9.88 

26.667 

EV 

2 

53.34 

4.000 

EM 

3 

12.00 

1.088 

EV 

1 

1.88 

0.667 

EM 

1 

8.67 

NTDS  Operatcr/EW  Supervisor: 


1.  TaiK/coCTunicate  with  the  SLQ-17  operator? 

2.  View  the  presentation  on  the  SLQ-17  console? 

3.  Talx/conisumcate  with  the  WLR-1  ooerator? 

4.  View  the  presentation  on  the  WLR-1  console? 

5.  View  the  WLR-i  Status  Board? 

6.  View  the  NTDS  Status  Boarc? 

7.  jpdate  the  NTDS  Status  Board? 

8.  Check  (visually)  the  SLQ-17  computer? 

9.  Reboot,  reset,  or  work  with  tne  SLQ-17  computer9 
18.  Check  MUTE? 

11.  Change  any  settings  on  MUTE? 

12.  CoiMunicate  outside  the  Module? 


45.667 

IC 

3 

136.39 

65.667 

EV 

3 

197.80 

72.333 

IC 

3 

215.99 

60.667 

EV 

3 

182.88 

45.008 

EV 

3 

135. 88 

48.333 

EV 

3 

i44.99 

16.080 

EM 

3 

48.88 

34.667 

EV 

1 

34.67 

4.  -880 

EM 

1 

4.88 

6.667 

EV 

1 

6.67 

4.800 

EM 

1 

4.89 

35.333 

EC 

3 

;85. 99 

KEY:  IC  -  Internal  Communications;  EC  -  External  Communications; 
EV  -  External  Visual;  EM  -  External  Manual 
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little  requirement  for  manual  links  outside  of  ones  own 
position.  An  external  communications  link  example  is  the 
link  between  the  EW  Supervisor/NTDS  operator  and  the 
communications  that  enable  him  to  communicate  outside  the 
Module.  However,  this  requires  that  the  operator  rise  from 
his  seat  to  communicate.  As  a  remedy,  the  NTDS  operator 
uses  a  hand  mike  that  hangs  down  near  his  console.  This  is 
a  partial  solution  because  he  still  needs  to  rise  from  his 
seat  to  select  another  station  on  the  communication  box. 
Additionally,  the  hand  mike  hanging  so  close  to  his  console 
presents  a  clutter  problem. 

Note  that  the  communication  and  visual  link  between 
the  NTDS  operator  (EW  Supervisor)  and  the  WLR-1  operator  is 
the  longest  and  partially  blocked.  The  links  between  the 
WLR-1  operator  and  the  NTDS  operator  and  SLQ-17  operator  are 
long,  allowing  him  to  view  very  little  of  the  environment. 
The  WLR-1  operator's  visual  links  are  very  long  and  the 
parallax  effect  severely  degrades  his  observation.  Note  the 
long  link  lines  between  the  SLQ-17  and  NTDS  positions  and 
the  WLR-1  Status  Board,  and  the  WLR-1  operator  and  the  NTDS 
Status  Board.  Finally,  note  the  very  long  external  visual 
links  to  the  SSQ-82  MUTE  and  that  they  cross.  MUTE  is 
required  to  be  checked  periodically  for  faults  or  changes  in 
the  various  monitor  boxes.  The  distance  is  great  enough 
between   MUTE  and  the  rest  of  the  Module  that  only  the  WLR-1 


55 


operator  can  effectively  monitor  it.   However,  this  requires 
considerable  movement  on  the  part  of  the  WLR-1  operator. 
2.  Link  Analysis  Table 

The  frequency  of  the  various  links  were  determined  by 
the  Link  Analysis  Questionnaire.  The  ideal  way  to  determine 
link  frequency  is  to  count  the  actions/link  interactions 
during  the  watch.  Since  this  was  not  possible,  the 
questionnaire  approach  was  chosen.  The  Link  Analysis  is 
intended  here  to  focus  attention  at  the  links  that  are  used 
most  often.  The  frequency  of  link  usage  is  multiplied  by 
the  weight  (criticality )  of  the  link  and  an  indication  of 
its  relative  importance  is  determined. 

When  the  links  associated  with  the  WLR-1  operator 
are  considered,  it  can  be  noted  the  longest  links  are  the 
internal  communication  and  external  visual  links  between  him 
and  the  SLQ-17  and  NTDS  positions.  These  links  are  also  the 
most  critical  and  the  most  frequently  used.  The  average 
number  of  times  the  operator  tries  to  view  the  NTDS  console 
is  73.333.  Yet  this  console  is  the  furthest  away  (see 
Figures  4  and  5) .  The  WLR-1  operator  communicates  more  with 
the  NTDS  operator  for  two  reasons.  Many  times  the  NTDS 
operator  is  also  the  EW  Supervisor.  The  fullest  picture  of 
the  entire  environment  of  surface,  subsurface,  and  air 
contacts  is  present  on  the  NTDS.  The  other  large  frequency 
usage  is  the  visual  links  for  the  presentation  on  the  SLQ-17 
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console.   This  console  is  only  slightly  closer  than  the  NTDS 
console. 

In  the  case  of  the  SLQ-17  operator,  the  first  six 
entries  in  Table  3  are  the  ones  with  the  greatest 
criticality  and  the  highest  frequency  of  use.  The  high 
criticality  and  frequency  associated  with  checking  the  SLQ- 
17  computer  is  understandable  since  the  SLQ-17  operator  is 
specifically  trained  to  know  what  to  look  for  on  the 
computer  face.  Note  that  the  SLQ-17  operator  views  the 
presentation  at  the  NTDS  console  much  more  than  that  at  the 
WLR-1  position.  It  can  be  seen  from  Figure  5  that  these 
external  visual  links  between  the  SLQ-17  and  the  NTDS  are 
much  shorter  than  between  SLQ-17  and  the  WLR-1.  At  the  same 
time,  the  SLQ-17  operator  communicates  more  with  the  WLR-i 
operator  than  with  the  NTDS  operator.  This  suggests  that 
the  SLQ-17  operator  gets  a  better  picture  of  the  environment 
from  the  NTDS  but  better  information  concerning  the 
environment  from  the  WLR-1. 

The  NTDS  operator/EW  Supervisor  are  combined  because 
many  times  the  EW  Supervisor  will  man  the  NTDS  console  for  a 
major  portion  of  the  watch.  This  is  necessary  because  all 
the  external  communications  are  at  or  near  the  NTDS  console. 
Note  the  large  frequency  and  high  criticality  associated 
with  communications  outside  the  Module  (external 
communications  link) .  There  appears  to  be  a  reversal  of 
interaction   between  the  NTDS  operator/EW  Supervisor  and  the 
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WLR-1  and  SLQ-17  positions.  He  views  the  SLQ-17  console 
more  than  communicates  with  the  operator  but  talks  more  to 
the  WLR-1  operator  than  views  the  WLR-1  displays.  Recall 
from  Figures  4  and  5  that  both  the  internal  communications 
and  the  external  visual  links  between  NTDS  and  WLR-1  are 
very  long.  Additionally,  note  how  much  he  looks  at  the  WLR- 
1  Status  Board  even  though  it  is  the  furthest  away  (Figure 
6)  . 

The  Link  Analysis  is  important  since  it  serves  to 
indicate  which  links  are  long,  important,  and  possibly 
overworked.  As  such  it  can  be  used  as  a  starting  point  in 
the  redesign  of  a  layout  by  showing  which  links  need  to  be 
reduced  in  length.  The  Link  Analysis  results  should  also 
support  the  results  of  the  Operability  Analysis. 

B.   OPERABILITY  ANALYSIS 

The  Operator  Subtask  Questionnaire  was  divided  into   two 
parts:   the  Subtask  Assessment  and  the  Ranking  Matrix.    The 
Subtask  Assessment  was  given  first.    The  criteria  for  this 
evaluation  and  the  test  itself  are  given  in  Appendix  A. 

The  second  half  of  the  Operator  Subtask  Questionnaire 
was  the  Ranking  Matrix.  All  returned  valid  rankings  (n=5) 
were  entered  into  the  matrix  and  a  mean  determined  for  each 
block  and  the  matrix  numbered  accordingly.  The  standard 
deviation  was  calculated  in  case  of  a  tie.  This  matrix  with 
the    mean   rank   values   and   the   standard   deviation    is 


58 


illustrated  in  Table  4.    The  resultant  rank  matrix  is  shown 
in  Table  5. 

Next  this  rank  ordering  was  converted  to  an  interval 
scale.  This  was  done  by  reversing  the  order  of  the 
numbering  so  that  the  best  of  the  Operator  Workload  and 
Space  Effectiveness  is  #16  and  the  worst  is  #1  (see  Table 
6) .  Using  this  as  a  base,  the  delta  method  of  linear 
expansion  was  used  to  determine  an  interval  scale.  See 
Appendix  C  for  a  brief  description,  example  of  the  delta 
method,  and  the  final  work  sheet  for  this  application. 

Table   7  shows  the  result  of  the  delta  method  which  is  the 
desired   interval  scale.    The  results  of  the   delta   method 
were   normalized   by  dividing  all  the  blocks  by  the   highest 
value  in  the  block;  in  this  application  it  was  102.   Table  8 
is  the  normalized  interval  scale  for  this  application. 

The  Operability  Value  was  weighted  (multiplied)  by  the 
mean  assessed  Criticality  of  that  particular  Subtask  to 
derive  the  Weighted  Operability  Value.  The  Weighted 
Operability  Value  has  the  potential  to  range  from  an 
absolute  minimum  of  0  (0x1)  to  an  absolute  maximum  of  500 
(100x5).   The  range  noted  was  14.14  to  418.87. 

The  Weighted  Deficit  Value  gives  an  indication  of  how 
much  improvement  is  needed  to  optimize  Layout  Effectiveness 
for  a  particular  Subtask.  The  greatest  Weighted  Deficit 
Value   was  485.00  while  the  least  was  28.09.    The   Weighted 
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TABLE   4.   MEAN  RANK  ORDER  AND  STANDARD  DEVIATION 
FOR  EACH  RATING  MATRIX  CELL 


1  9.6  1  4.6  1  2.4  1  1  1 
1       2.50     1        1.14     1       0.55     1                   1 

1  11.0  1  8.0  1  5.2  1  2.8  1 
1       2.55     1        1.58     1        1.10     1     0.84     1 

1  13.8  1  10.8  1  8.8  1  5.4  1 
1        1.64     1        1.10     1        1.10     1     1.34     1 

1  16  1  14.0  1  12.2  1  10.4  1 
1                      1        1.22     1       2.17     1     3.13     1 

I  I 

I  Mean  Rank  I 

I  Standard  Deviation  I 


I 


TABLE  5.  RANK  ORDER  OF  OPERATOR  RATING  MATRIX 


SE 


1           9 

4 

2 

1           1 

1        12 

7 

5 

3           1 

1        14 

11 

8 

6           1 

1        16 

15 

13 

10          1 

2        3 
OW 
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TABLE  6.  FINAL  RANK  ORDER  INVERTED  FOR  DELTA  METHOD 


SE 


SE 


1     8 

13 

15 

16   1 

1    5 

10 

12 

14   1 

1    3 

6 

9 

11   1 

1    1 

2 

4 

7    1 

2       3 

OW 


TABLE   7.   DELTA  METHOD  SOLUTION  FOR 
OPERATOR  SUBTASK  RATING  SCALE 


22 


35 


47 


1    55 

77 

90 

102   1 

1    40 

62 

75 

87   1 

1    24 

46 

59 

71    1 

1      0 

22 

35 

47   1 

55 


40 


24 


2       3 

OW 
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TABLE  8.  NORMALIZED  INTERVAL  SCALE 


SE 


21      34 


2       3 
OW 


46 


1     54 

75 

89 

100   1 

1     40 

61 

74 

86   1 

1     24 

45 

58 

70   1 

1      0 

21 

34 

46   1 

54 


40 


24 
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Deficit  Value  can  range  from  500  to  0.  The  larger  the 
number,  the  greater  the  amount  of  improvement  is  needed. 

The  Total  Module  Operability  for  this  particular  EW 
Module  was  computed  to  be  39.2  X.  This  computation  is  as 
follows.  There  were  49  Subtask  evaluated.  The  summation  of 
the  criticalities  of  these  Subtasks  in  order  to  accomplish 
the  mission  was  200.84.  By  assuming  a  perfect  layout,  we 
can  multiply  by  100  to  obtain  a  maximum  score  of  20,084. 
Next  the  Weighted  Operability  Values  were  summed  to  obtain 
the  actual  score  of  the  Module  of  7872.31.  When  the  actual 
score  is  divided  by  the  maximum,  an  indication  of  the 
effectiveness  of  the  layout  is  obtained. 

Table  9  contains  an  ordering  of  the  Subtasks  by 
cumulative  weight.  This  was  determined  by  dividing  the 
Weighted  Deficit  Value  by  the  optimum  layout  effectiveness 
to  determine  how  much  the  deficit  each  Subtasks  comprises. 
These  were  then  ranked  from  most  to  least.  This  table  gives 
an  indication  of  which  Subtasks  should  be  improved  first  in 
order  to  achieve  the  most  cost  effective  approach  to 
improving  the  Module. 

Table  9  contains  the  rank  ordering  by  cumulative 
weights,  the  Subtask  number,  a  brief  description  of  the 
Subtask  and  its  associated  position,  the  operators  polled 
with  their  evaluation  of  the  Subtask  in  terms  of  Operator 
Workload  and  Space  Effectiveness  converted  to  an  interval 
scale,   the   operability   value  (the  mean  of  the   operators' 
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TA3LE    9.       RANK 

ORDER    OF 

SUB1 

"ASKS    BY    CUMULI 

lTivh 

WEIGH 

Operapility 

Deficit 

Cueeslative 

to.  Subtask 

Dwcnotion  (Pos) 

oi  03 

03    04 

03    06 

07 

Value 

Value     Critical itv 

Ueiaht 

Total* 

1      1.2,1 

Rprt  EU  Info  (Sup) 

21      1 

8     8 

0       8 

3 

1888 

97.888 

5.000 

197 

197 

2     1.2.2 

Prov  SI  Recce  (Supi 

21      1 

8     8 

8     8 

3 

1088 

97.388 

3.380 

197 

7.94 

3     4.1.1 

Sreh  AssqnBnds  (UUU 

8     8 

8     8 

21      8 

3 

1088 

97.888 

3.888 

197 

11.91 

*     4.2.2 

Rprt  EU  Irrfo  (UUU 

34     8 

8     8 

8   21 

3 

7.337 

32.143 

3.088 

177 

15.53 

3     4.2.1 

Pry  at  -  KTDS  (UUU 

21     8 

8     8 

8     8 

3 

1888 

97.088 

4.714 

174 

19.42 

6     4.1.4 

OF  Signals  (UUU 

21     8 

8     8 

SB     8 

3 

11.206 

88. 714 

3.088 

163 

2183 

7     4.1.2 

Analyre  Sigs  (UUU 

8     8 

8     8 

21     8 

3 

14.286 

35.714 

1008 

151 

26.56 

8     3.1.1 

Co— licate  (SB) 

21    24 

8    21 

8     8 

21 

18.373. 

39.427 

4.371 

133 

29.31 

9     2.1.1 

Enter  ID'S  (NTDS) 

43    21 

8      8 

8     8 

8 

9.337 

98.143 

4.508 

132 

23.23 

11    4.2.4 

log  Intercepts  (UUU 

8     8 

8     8 

78    24 

3 

11429 

36.571 

4.371 

124 

36.47 

11    2.1.4 

Enter  EU  Fix  (NTDS) 

34     8 

8    21  1 

,88      8 

8 

22.143 

77.337 

1008 

119 

39.56 

12    2.1.2 

Enter  ESI  Info  (NTDS) 

43     8 

8    21 

8      8 

3 

9.857 

98.143 

4.714 

108 

42.74 

13    3.1.3 

Update  SB  (SB) 

43      8 

8    34 

21    21 

21 

28.289 

79.711 

4.714 

188 

45.32 

14    2.1.3 

Brno.  Resolution  (NTDS) 

sa  2i 

8    45 

21      8 

21 

21714 

76.286 

4.429 

177 

48.59 

13    11.3 

Trnqle  ESN  Bfnq  (NTDS) 

21     8 

8    21 

38      8 

45 

23.888 

75.888 

4.429 

172 

51.31 

16    1.1.  S 

Nan  ID  Request  (SUP) 

21    34 

8    34 

-    21 

— 

22.088 

78.088 

4.008 

156 

3137 

17    L  1.4 

Assign  E5N  Srcft  (Sup) 

58    34 

8    21 

S3     8 

3 

24.429 

73.571 

4.088 

148 

56.35 

18    3.1.2 

Advise  toaula  (SB) 

34    21 

21    21 

3      3 

74 

24.429 

75.371 

4.088 

148 

SB.  33 

19    2.2.1 

Report  EU  Info  (NTDS) 

43    61 

8    SB 

SB    21 

3 

34.714 

65.286 

4.571 

144 

61.27 

21    1.2.3 

Nay  by  ESN  (Sup) 

21    21 

34    21 

3    21 

74 

17.508 

32.508 

1598 

137 

6154 

21    3.2.4 

Update  SB  (17) 

21    45 

21    21 

21    21 

21 

24.357 

75.143 

1371 

128 

63.34 

22    1.1.3 

Assign  ESN  resp  (Sup) 

-    45 

34    45 

21    - 

73 

44.888 

56.388 

4.500 

136 

67.38 

23    1.1.3 

Nan  ID  Roust-sftip  (Supj 

l  58    34 

21    45 

-    46 

— 

48.388 

59.288 

4.258 

186 

69.36 

24    3.1.3 

Evaluate  Data  (17) 

45    45 

45    21 

188    21 

43 

46.888 

54.888 

4.571 

182 

71.39 

23    3.1.3 

Enter  Paraseters  (17) 

48    S4 

54    34 

21    54 

48 

45.286 

54.714 

4.429 

1.98 

7135 

25    4.3.4 

tonitor  NUTS  (UUU 

34    34 

34    34 

21    34 

SB 

33.371 

64.429 

1714 

1.96 

73.32 

27    1.2.4 

De/Brief  Aintinq  (Supi 

34    21 

34    21 

3    21 

74 

29.286 

78.714 

1286 

1.98 

77.32 

28    4.3.1 

tonitor  EKQt  (UUU 

SB    21 

61    S3 

21    61 

45 

46.429 

51571 

4.286 

1.38 

79.78 

29    4.3.2 

Rprt  SNCQN  (UUU 

53    34 

46    SB 

21    46 

46 

44.143 

55.337 

1357 

1.75 

31.46 

38    1.3.1 

EDI  Criteria  (Sup) 

58    59 

38    45 

21    45 

— 

52.588 

47.588 

4.590 

1.75 

3121 

31     1.1.2 

Assign  Searcn  (Sup) 

SB    45 

34    45 

21    78 

75 

49.714 

58.286 

4.143 

1.71 

34.32 

32    3.1.1 

tonitor  AutoCrrltn  (17)  61    73 

54    75 

8    75 

24 

32.888 

48.088 

4.286 

1.59 

36.61 

33    2.1.6 

Evai  ESN  8mq  (NTDS) 

78    74 

74    SB 

21    45 

5B 

37.143 

42.357 

4.286 

L58 

38.11 

34    2.2.2 

Update  SB  (NTDS) 

45    74 

74    SB 

21    45 

45 

31.714 

48.286 

1371 

1.41 

39.52 

33    4.3.3 

Log,  Violations  (UUU 

SB    S3 

34    46 

78    34 

45 

49.2B6 

58.714 

1808 

1.25 

98.77 

36    1.2.3 

Update  SB  (Sup) 

78    SB 

S3    SB 

S3    SB 

61 

68.143 

39.357 

1714 

1.21 

91.39 

37    12.1 

Report  EU  (17) 

86  188 

36    36 

21    36 

3 

66.429 

31571 

4.286 

1.18 

9116 

38    112 

Provide  Data  (17) 

36    86 

74    74 

S3    78 

24 

67.429 

32.571 

4.800 

1.87 

94.23 

39    112 

EstaP  EDI  Nodes  (17) 

78  188  188    86 

45  108 

3 

71.714 

28.286 

4.371 

1.86 

33.29 

4*    4.1.3 

Assist  Eval  EDI  (UUU 

34    34 

34  108 

34    46 

SB 

48.571 

51.429 

1286 

3.96 

96.25 

41    11.2 

EstaP  Op  Modes  (17) 

86  188  108  188 

53    73 

45 

38.571 

19.714 

1357 

3.51 

97.36 

42    1.1.1 

Assign  SUM7  (Supi 

78  188  188  188 

SB    SB 

73 

38.143 

19.857 

1167 

8.52 

97.33 

43    113 

Assist  EDI  Esploy  (17) 

SB  188  108  188 

SB  108 

38 

32.286 

17.714 

1333 

3.48 

97.36 

44    113 

tonitor  EU  Entry  (17) 

78  188 

36    74 

46  108 

74 

78.571 

.    21.429 

1571 

3.45 

98.31 

45    111 

Engage  Trgts-ED!  (17) 

74  188 

36    74 

188  188 

38 

38.357 

11.143 

4.714 

3.43 

38.74 

46    11.4 

tonitor  Envrnant  (17) 

86  188  188  188 

36  108 

45 

38.143 

11.357 

4.808 

3.39 

99.13 

47    4.13 

Update  SB  (UUU 

SB  108  188    36 

SB  108 

74 

32.286 

17.714 

1429 

8.33 

99.48 

48    1.1.7 

tonitor  SLfl-17  (Sup) 

36  108  108  108 

78    36 

— 

98.333 

9.667 

1667 

8.29 

99.77 

49    4.1.3 

Check  (sagas  (UUU 

78  108  188  188 

SB  188 

36 

37.714 

12.286 

1286 

3.23 

108. 88 
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©valuations),  the  deficit  value  (100  -  operability  value), 
the  mean  criticality,  the  cumulative  weight  or  percentage  of 
the  total  deficit  that  that  particular  Subtask  comprises, 
and  the  total  percentage. 
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V.  DISCUSSION  AND  RECOMMENDATIONS 

There  has  been  no  attempt  to  ascertain  what  Weighted 
Deficit  Value  or  Weighted  Operability  Value  is  acceptable. 
This  is  beyond  the  scope  of  this  effort.  The  purpose  has 
been  to  identify  which  areas  are  in  need  of  improvement  and 
what  areas  should  be  addressed  first  in  order  to  realize  the 
greatest  amount  of  improvement  for  a  given  effort.  To 
answer  the  question  of  what  Weighted  Deficit  or  Operability 
Value  is  acceptable  will  call  for  additional  research 
targeting  the  Subtasks  individually  to  a  greater  detail  than 
was  attempted  here. 

A.   LINK  ANALYSIS  DISCUSSION 

The  Link  Analysis  results  show  that  there  is  only  one 
position  that  might  be  considered  acceptable  in  relation  to 
the  lengths  of  its  links.  This  is  the  SLQ-17  position. 
This  can  be  seen  in  part  from  the  relatively  good  showing 
that  the  SLQ-17  console  position  had  in  comparison  to  the 
other  two  positions.  The  SLQ-17  operator  can  easily  view 
what  is  displayed  on  the  NTDS  console  and,  without  excessive 
movement,  view  the  WLR-1  displays.  He  is  within  good 
viewing  distance  of  the  NTDS  Status  Board  and  his  own  SLQ-17 
computer.  The  viewing  distance  to  the  WLR-1  position  and 
its  associated  Status  Board  are  rather  long,  but  still 
viewable.    Because   of   its  relatively  good  positioning   in 
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relation  to  the  rest  of  the  Module,  SLQ-17  entries  were  much 
lower  in  Table  9.  This  would  indicate  that  the  layout 
actually  promotes  increased  operator  compensation  since  the 
other  positions  did  not  score  as  well.  A  score  of  39.2%  is 
an  indication  of  a  poor  the  Module  layout  contributing  to  an 
increased  operator  compensation  burden.  Were  the  module 
layout  better,  the  operators  would  have  felt  much  better 
about  the  Module  and  the  score  would  have  been  higher. 

B.   OPERABILITY  ANALYSIS  DISCUSSION 

Several  observations  can  be  made  from  Table  9.  First, 
the  SLQ-17  appears  to  be  the  best  position  in  the  EW  Module 
since  its  first  entry  is  in  twenty-first  place  in  the  table 
and  most  of  the  entries  are  at  the  bottom  of  the  table. 
Almost  27*  of  the  possible  improvements  can  be  made  in  the 
first  seven  entries  and  these  are  just  for  the  EW  Supervisor 
and  the  WLR-1  operator.  Note  that  the  criticalities  of 
these  Subtaaka  are  the  highest.  In  other  words,  these 
Subtasks  which  are  very  critical  are  poorly  supported  by  the 
layout,  relative  to  the  less  critical  Subtasks.  Most  of  the 
lower  criticalities  are  associated  with  Subtasks  that  have  a 
relatively  good  layout  effectiveness. 

It  can  be  reasonably  argued  that  Module  Operability  of 
39. 25*  is  not  sufficient  for  an  EW  Module.  What  can  not  be 
argued  is  how  much  improvement  is  enough.  Nor  can  it  be 
extrapolated     from    this    study    what    improvement    a 
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rearrangement,  can  result,  in.  However,  it  can  be  seen  that 
improvement  can  be  made  in  certain  areas,  as  is  indicated  by 
careful  perusal  of  Table  9. 

C.  EXTRAPOLATION 

Further,  this  approach  can  be  used  for  possible 
extrapolations.  For  example,  comparing  Figure  1  and  Figure 
2,  similarities  are  noted.  They  have  the  same  arrangement 
of  positions  (i.e.,  from  left  to  right,  WLR-1,  SLQ-17,  and 
NTDS) .  The  positions  are  arranged  in  "straight  line"  type 
of  layout.  This  resulted  in  a  low  Layout  Effectiveness 
rating  for  USS  CARL  VINSON.  It  may  be  readily  conjectured 
that  another  arrangement  would  work  better,  namely,  a 
"crescent"  shaped  layout  with  the  NTDS  in  between  WLR-1  and 
SLQ-1  and  the  supervisor's  position  raised  and  directly  in 
back  of  the  NTDS  operator. 

D.  CONCLUSION 

It  can  be  concluded  that  the  present  configuration  of 
the  EW  Module  on  USS  CARL  VINSON  does  not  result  in  an 
optimal  utilization  of  this  Module  in  terms  of  EW  mission 
accomplishment.  Further,  there  is  a  real  need  to  assess  the 
layout  operability  of  the  warfare  modules  onboard  U.S.  Naval 
combatants.  This  thesis  has  provided  one  way  in  which  a 
measure   of  the  effectiveness  of  a  particular  layout  can   be 
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determined.  Although  this  was  a  limited  teat,  indicatlona 
are  that  thia  approach  works,  and  further  testing  is 
warranted. 

Building  a  new  layout  is  urged  with  the  hopes  that  it 
may  prove  by  testing  to  be  better  than  the  last  one,  using 
the  Link  Analysis  and  Operability  Analysis  illustrated  in 
this  work.  What  is  significant  and  useful  from  the  Link 
Analysis  is  that  any  improvement  in  layout  design  should 
probably  start  with  ensuring  that  the  critical  links  are  not 
overly  long  or  taxed  beyond  their  limit.  Any  improvement  to 
the  layout  design  should  take  into  account  these  critical 
links  to  reduce  them  to  their  optimum  and  any  changes  must 
not  adversely  affect  the  links  since  in  that  case  any  gain 
in  layout  design  may  be  cancelled  by  a  loss  in  link 
utilization.  By  conducting  tests  at  landbased  test  sites, 
the  risks  of  error  are  reduced.  By  the  utilization  of  mock- 
ups  and  fleet  inputs,  the  risks  can  be  reduced  even  further. 
The  result  is  a  more  effective  layout  enhancing  mission 
accomplishment . 

E.   RECOMMENDATION 

It  is  recommended  that  a  land  based  test  facility  be 
established  that  would  incorporate  the  ideas, 
recommendations,  and  methods  indicated  in  this  thesis  as  a 
first  step  to  upgrading  our  combat  workspaces. 
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APPENDIX  A 

OPERATOR  SUBTASK  QUESTIONNAIRE 
GENERAL  INSTRUCTIONS 
The  purpose  of  this  questionnaire  is  a  subjective 
evaluation  of  the  layout  effectiveness  of  the  EW  Module  for 
use  in  an  algorithm  to  determine,  in  an  objective  sense,  the 
effectiveness  of  the  layout  in  accomplishing  the  mission  of 
the  Module.  To  do  this  there  is  a  series  of  subtasks 
differentiated  by  operator  that  must  be  assessed  in  terms  of 
operator  workload  per  subtask,  space  effectiveness  per 
subtask,  and  criticality  of  the  subtask  toward  overall 
mission  accomplishment.  What  is  required  is  to  make  this 
assessment  based  on  your  experience  and  expertise.  There  is 
no  time  limit,  you  may  ask  questions  of  anyone  you  wish,  and 
you  should  go  and  look  at  the  Module  to  make  sure  of  your 
answers  especially  if  you  are  unsure  of  some  of  the 
questions  concerning  movements.  There  are  no  right  or  wrong 
answers,  but  try  to  be  as  precise  as  you  can.  A  scenario, 
hopefully  similar  to  your  recent  operations  in  the  Sea  of 
Japan,  has  been  constructed.  For  each  of  the  subtasks  on 
the  next  page,  mark  with  an  "X",  the  description  that  best 
describes  the  operator  workload  (OW)  and  the  space 
effectiveness  (SE) .  If  the  arrangement  of  the  space  has 
little  or  no  effect  on  subtask  accomplishment,  then  it  would 
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rate  the  highest  (4) .  Conversely,  if  the  layout  or 
arrangement  of  the  space  negatively  impacts  subtask 
accomplishment,  then  it  would  rate  a  (1).  Give  your 
assessment  of  the  criticallty  of  the  subtask  in  relation  to 
the  overall  mission  accomplishment.  The  descriptions  of 
criticality,  operator  workload,  and  space  effectiveness  are 
listed  on  a  separate  sheet. 

SCENARIO 

This  scenario  begins  with  the  assumption  of  the  watch 
by  a  particular  section 

They  are  the  on-coming  watch  section  in  the  EW  module 
of  a  NIMITZ  class  aircraft  carrier  that  is  steaming  in  the 
open  ocean  with  six  escorts.  The  escorts  are  one  VIRGINIA 
class  cruiser,  two  SPRUANCE  class  destroyers,  one  OLIVER 
HAZARD  PERRY  class  frigate,  one  LOS  ANGELES  class  submarine, 
and  an  oiler.  There  are  heightened  tensions  world-wide  with 
a  probable  confrontation  between  the  two  super-powers. 
There  is  a  Soviet  task  group  within  200  NM .  The  task  group 
is  comprised  of  a  KIEV  class  aircraft  carrier,  a  KIROV  class 
cruiser,  a  SOVERMENYY  class  destroyer,  two  KRIVAK  III  class 
frigates,  a  SLAVA  class  destroyer,  and  three  auxiliaries. 
Additionally,  ECHO  II,  VICTOR  III,  and  OSCAR  class 
submarines  are  known  to  be  in  the  area  but  unlocated  for  the 
past   twelve  hours.    A  Mod-KASHIN  is  the  tattletale  for  the 


71 


Battle   Group.    Both  forces  are  within  range  of  Soviet   air 
power . 

General  Quarters  is  not  set,  but  a  heightened  Condition 
III  steaming  watch  is  manned.  There  has  been  a  momentary 
lapse  of  400  Hz  power  and  the  NTDS  is  being  reloaded.  The 
SLQ-17  needs  to  be  reloaded  and  reprogrammed.  As  the  NTDS 
is  brought  on  the  line,  the  WLR-1  operator  is  told  to 
recheck  the  past  entries  in  his  log  and  verify  that  they  are 
still  active.  After  15  minutes,  the  WLR-1  operator  reports 
that  he  has  intercepted  several  new  signals.  One  is  an 
airborne  mapping  and  reconnaissance  radar.  One  appeared  to 
be  a  brief  intercept  of  a  submarine  radar.  Another  is  an 
air  search  radar  and  the  last  is  a  missile  acquistion  radar. 

The  NTDS  air  trackers  report  jamming  on  both  long  range 
and  3-D  air  search  radars. 

The  SLQ-17  alarms  and  displays  hostile  missile  symbols 
from  both  the  suspected  direction  of  the  SOviet  task  group 
and  two  angles  30  degrees  either  side  of  the  task  group. 

Deck  Launched  Interceptors  are  airborne  within  one 
minute. 

The  EW  operator  at  the  NTDS  console  is  entering  ESM 
bearing  lines  and  attempting  to  identify  unknown  contacts. 
The  SLQ-17  operator  shifts  operation  of  the  ECM  portion  to 
automatic  as  the  TAO  frees  weapons.  The  WLR-1  operator  is 
attempting  to  search  the  known  hostile  missile  homing  radar 
ranges   to  facilitate  identification.    General  Quarters   is 
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sounded.  The  TAO  orders  EMCON  to  be  set  for  battle  and  the 
WLR-1  operator  selects  EMCON  D  on  MUTE.  A  quick  check  of 
both  the  NTDS  scope  and  that  of  the  SLQ-17  indicates  that 
the  number  and  direction  of  the  inbound  unknowns  do  not 
match.  The  EW  watch  section  tries  to  match  the  emerging 
identification  from  the  WLR-1  and  SLQ-17  to  both  the  SLQ-17 
and  the  NTDS  presentations. 

If  you  have  trouble  envisioning  this  scenario,  recall 
the  Sea  of  Japan  operations  on  your  last  deployment  and 
consider  the  signal  environment  and  tactics  you  saw  then. 


73 


OPERATOR  WORKLOAD.  SPACE  EFFECTIVENESS,  and  CRITICALITY 
WORKLOAD/COMPENSATION/INTERFERENCE  (Mental  and  Physical) 


(1) 

Workload  Extrese 
Compensation  Extreme 
Interference  Extreae 


(2) 

Workload  High 
Coapensation  High 
Interference  High 


(3) 
Workload  Moderate 
Coapensation  Moderate 
Interference  Moderate 


(4) 
Workload  Low 
Coapensation  Ion 
Interference  Low 


SPACE  EFFECTIVENESS 


(1) 
Inadequate  Performance 
Due  to  Layout 


(2) 
Adequate  Performance 
Achievable;  Layout 
Sufficient  to  Spec- 
ific Task 


(3) 
Layout  Enhances  Spe- 
cific Task  Accomplish- 
ment 


(4) 

Layout  Design 
Integrates 
Multiple  Tasks 


CRITICALITY:    How    important    is    it    that    the    operator/team    be 
able         to      perform       this      task      as      compared       to      the      other 
tasks    in    successfully    completing    the    mission? 

Scale    Value: 

<1)   Of  very  small  importance.    Ability  to   perform   this 
task  as  compared  to  other  tasks  in  this  duty  is  unimportant, 
or  almost  unimportant,  in  order  to  successfully  complete  the 
mission  of  the  Module. 

(2)   Of  small  importance.    This  task  within  this  duty   is 
less   important   than   most  tasks  required   to   successfully 
complete  the  mission  of  the  Module. 

<3)  Of  moderate  importance.   This  task  within  this  duty  is 
about   as  important  as  most  tasks  required   to   successfully 
complete  the  mission  of  the  Module. 

<4)  Of  substantial  importance.   This  task  within  this  duty 
is   more  important  than  most  tasks  required  to   successfully 
complete  the  mission  of  the  Module. 

(5)  Of  extreme  importance.    This  task  within  this  duty  is 
extremely   important   in  order  to  successfully  complete   the 
mission  of  the  Module. 
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EW  Supervisor 

Task:  1.1  Direct  ESM  search 

Subtaaka 

1.1.1  Assign  search  parameters  to  SLQ-17 

Operator  Workload:   _ 


<1)      (2)      (3)      (4) 
Space  Effectiveness: 


<1)     (2)     <3)     (4) 

Criticality:     

1.1.2  Assign  search  parameters  to  WLR-1 

Operator  Workload:   


<1)      (2)      (3)      (4) 
Space  Effectiveness: 


(1)     (2)     (3)      (4) 

Criticality:     

1.1.3  Assign  ESM  sensor  report  responsibilities   --  own  ship 

Operator  Workload:   

<1)      (2)      <3)      (4) 

Space  Effectiveness: 


<1)      (2)      (3)      (4) 

Criticality:     

1.1.4  Assign  ESM  sensor  report  responsibilities   --  force 

Operator  Workload:   

(1)      (2)      <3)      (4) 

Space  Effectiveness: 


(1)      (2)      (3)      (4) 
Criticality:     
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1.1.5  Initiate  manual  ID  request  -  ship 
Operator  Workload:   . 


(1)     <2>     <3)     (4) 
Space  Effectiveness: 


(1)     (2)     (3)     (4) 

Criticality:     

1.1.6  Initiate  manual  ID  request  -  force 

Operator  Workload:   


<1)     <2)     (3)     (4) 
Space  Effectiveness : 


<1)     (2)     (3)     (4) 

Criticality:     _____ 

1.1.7  Monitor  automatic  correlations/associations,  (SLQ-17) 

Operator  Workload: 

(1)      <2)      <3)      <4) 

Space  Effectiveness : 


<1)     <2)     <3)     (4) 
Criticality: 


Task:  1.2  Report/Disseminate  EW  Information 

Subtaaka 

1.2.1  Report  evaluated  EW  information 


Operator  Workload: 


<1)     (2)     (3)     (4) 
Space  Effectiveness: 


<1)      (2)      <3)      (4) 

Criticality: 
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1.2.2  Provide  EW  recommendations 
Opera-tor  Workload:   


(1)     <2>     (3)  (4) 

Space  Effectiveness: 

(1)     <2>     <3>  <4) 

Criticality:     

1.2.3  Update  status  board  near  NTDS  console 

Operator  Workload:   


<1)     (2)  <3)  (4) 

Space  Effectiveness: 

<1)     (2)  <3)  (4) 

Criticality:     

1.2.4  Brief /debrief  embarked  Airwings 

Operator  Workload:   


(1)     (2)  (3)  (4) 

Space  Effectiveness: 

<1)     <2)  (3)  <4) 

Criticality:     

1.2.5  Navigation  by  passive  EW 

Operator  Workload:   


(1)  (2)     (3)  (4) 

Space  Effectiveness: 

<1)  (2)     (3)  (4) 

Criticality:     


Task :  1.3  Counter  Hostile  Environment 
Subtasks 
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1.3.1  Promulgate  ECM  employment  criteria 
Operator  Workload:   _____   _ 


(1)     <2>     <3>     (4) 
Space  Effectiveness:, ______ 


<1)     (2)     <3)     (4) 

Criticality:     


NTDS  Operator 

Task:  2.1  Collect  and  enter  EW  data  into  NTDS 

Subtasks 

2.1.1  Enter  manual  ID  information  into  NTDS 

Operator  Workload:   _____   


<1)     <2)     (3)     (4) 
Space  Effectiveness :        _____ 


(1)     (2)     <3)     (4) 

Criticality:     

2.1.2  Enter  manual  ESM/NTDS  track  associations 

Operator  Workload: , 


<1)     (2)     (3)     <4) 
Space  Effectiveness: 


<1)     (2)     (3)     (4) 

Criticality:     

2.1.3  Perform  triangulation  of  ESM  bearing . lines 

Operator  Workload:   


<1)      <2)      (3)      (4) 
Space  Effectiveness: ______   _____ 


(1)     (2)     (3)     (4) 
Criticality:     
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2.1.4  Enter  EW  fixes 

Opera-tor  Workload: 


(1)     (2)     <3>     <4> 
Space  Effectiveness: 


(1)      (2)      (3)      <4) 

Criticality:     

2.1.5  Advise  operators  of  bearing  resolution 

Operator  Workload:   


<1)      (2)      (3)      <4> 
Space  Effectiveness: 


<1)     <2)  (3)      (4) 

Criticality:     

2.1.6  Evaluate  externally  reported  ESM  bearings 

Operator  Workload:   


(1)     (2)     (3)     (4) 
Space  Effectiveness: 


(1)     (2)     (3)     (4) 

Criticality:     

Task :  2.2  Report/Disseminate  EW  Information 

Subtask 

2.2.1  Report  evaluated  EW  information 

Operator  Workload:   


(1)     (2)     (3)     (4) 
Space  Effectiveness: 


<1)      (2)      <3)      (4) 

Criticality:     

2.2.2  Update  status  board  near  console 

Operator  Workload:   


<1)      (2)      (3)      (4) 
Space  Effectiveness: 


<1)      (2)      (3)      <4) 
Criticality:     
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SLQ-17  Operator 

Task :  3 . 1  Conduct  ESM  Search 

Subtaak 

3.1.1  Monitor  automatic  correlations/ associations 

Operator  Workload:   


(1)      (2)      (3)      <4> 
Space  Effectiveness: 


<1>      <2)      <3)      <4) 

Criticality:     

3.1.2  Establish  operating  modes  of  SLQ-17  (ESM) 

Operator  Workload: 


<1)      <2)      <3)      (4) 
Space  Effectiveness : ,   


(1)     (2)     (3)     (4) 

Criticality:     

3.1.3  Enter  detection  and  response  parameters  (ESM/ECM) 

Operator  Workload:   

(1)     <2)     (3)     (4) 

Space  Effectiveness: ___^   


(1)     (2)     (3)     (4) 

Criticality:     

3.1.4  Monitor  environment  on  NTDS  console 

Operator  Workload:   


<1)     (2)     (3)     (4) 
Space  Effectiveness: 


(1)      (2)      (3)      (4) 

Criticality:     


80 


3.1.5  Evaluate  displayed  data 
Operator  Workload:   


(1)     <2>     (3)  (4) 

Space  Effectiveness: 

<1)     <2>  <3)  (4) 

Criticality:     


Task:  3.2  Report/Disseminate  EW  Information 

Subtask 

3.2.1  Report  evaluated  EW  information 


Operator  Workload: 


<1)     (2)  (3)  (4) 

Space  Effectiveness: 

<1)     (2)  (3)  (4) 

Criticality:     

3.2.2  Provide  ESM/ECM  data  to  NTDS 

Operator  Workload:   


<1)     (2)     (3)  (4) 

Space  Effectiveness: 

<1)     (2)     (3)  (4) 

Criticality:     

3.2.3  Monitor  entry  of  EW  data  into  NTDS 

Operator  Workload:   


<1)  (2)      (3)  (4) 

Space  Effectiveness: 

<1>  (2)     (3)  (4) 

Criticality:     
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3.2.4  Update  status  board 
Operator  Workload: 


3.3.1  Engage  targets  with  ECM 
Operator  Workload:   


<1)     <2)     (3)     (4) 
Space  Effectiveness: 


(1)     (2)     <3>     (4) 

Criticality:     


Task :  3.3  Counter  Hostile  Environment 
Subtask 


<1>     (2)     (3)     (4) 
Space  Effectiveness :        _ 


<1)     <2)     (3)     <4) 

Criticality: 

3.3.2  Establish  ECM  operating  Modes 

Operator  Workload:   


(1)     (2)     (3)     (4) 
Space  Effectiveness : 


<1)     (2)     (3)     (4) 

Criticality:     

3.3.3  Assist  in  promulgation  of  ECM  employment  criteria 

Operator  Workload:   

<1)      (2)      (3)      (4) 

Space  Effectiveness: 


<1)     (2)     (3)     (4) 
Criticality:     ______ 
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WLR-1  Operator 

Task:  4 . 1  Conduct.  ESM  Search 
Subtasks 

4.1.1  Search  assigned  bands 

Operator  Workload:   


<1>      (2)  (3)  (4) 

Space  Effectiveness: 

<1)     (2)  (3)  (4) 

Criticality:     

4.1.2  Analyze  intercepted  signals 

Operator  Workload:   


<1>     (2)     (3)  (4) 

Space  Effectiveness: 

<1)     (2)     (3)  <4) 

Criticality:     

4.1.3  Check  intercepts  for  images/harmonic3 

Operator  Workload:   


<1>     <2)     (3)  <4) 

Space  Effectiveness: 

(1)     (2)     (3)  (4) 

Criticality:     

4.1.4  Accurately  DF  intercepted  signals 

Operator  Workload:   


<1)     (2)     (3)  (4) 

Space  Effectiveness: 

<1)     (2)     (3)  (4) 

Criticality:     
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4.1.5  Assist  in  evaluating  ECU 
Operator  Workload:   „ 


(1)     (2)     (3)     <4) 

Space  Effectiveness: 

(1)     <2)     <3)     (4) 

Criticality:     

Task:  4.2  Report/Disseminate  EW  Information 

Subtasks 

4.2.1  Provide  ESM  data  to  NTDS 

Operator  Workload:   


(1)     (2)     (3)  <4) 

Space  Effectiveness: _____  

(1)     (2)     (3)  (4) 

Criticality:     

4.2.2  Report  evaluated  EW  information 

Operator  Workload: 


(1)     (2)     (3)  (4) 

Space  Effectiveness: ,   

<1)     (2)     (3)  (4) 

Criticality:     

4.2.3  Update  status  board  near  position 

Operator  Workload:   


<1)  (2)     (3)  (4) 

Space  Effectiveness: ,  

(1)  (2)     (3)  (4) 

Criticality:     . 
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4.2.4  Log  all  intercepts 
Opera-tor  Workload: 


(1)      <2>      (3)      (4) 

Space  Effectiveness: 

(1)      <2>      (3)      (4) 

Criticality:     

Task:  4.3  EMCON 

Subtasks 

4.3.1  Monitor  EMCON 

Operator  Workload:   


(1)     (2)     <3>  <4) 

Space  Effectiveness: 

(1)     (2)     (3)  (4) 

Criticality:     

4.3.2  Report  violations  of  EMCON 

Operator  Workload:   


<1)     <2)  (3)  (4) 

Space  Effectiveness: 

(1)     (2)  (3)  (4) 

Criticality: 

4.3.3  Log  violations  of  EMCON 

Operator  Workload:   


(1)  <2>     (3)  (4) 

Space  Effectiveness: 

(1)  (2)     (3)  (4) 

Criticality:     

4.3.4  Monitor  MUTE 

Operator  Workload:   


(1)     (2)     (3)     (4) 

Space  Effectiveness: 

(1)      (2)      (3)      (4) 

Criticality:     
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EW  Status  Board 

Task,:  5.1  Maintain  Status  Boards 

Subtaaka 

5.1.1  Communicate  with  operators 

Operator  Workload:   , 


<1)      (2)      <3>      <4> 
Space  Effectiveness : 


<1)      (2)      <3)      (4) 

Criticality:     

5.1.2  Advise  operators  of  any  information  received 
Operator  Workload:   


<1)     (2)     (3)     <4) 
Space  Effectiveness: 


<1)     (2)     (3)     (4) 

Criticality: 

5.1.3  Update  status  boards 

Operator  Workload: _____ 


(1)      (2)      (3)      (4) 
Space  Effectiveness: 


(1)     (2)     (3)     (4) 

Criticality:     _____ 


86 


Rating    Matrix    Cell    Rank    Order 
For    the    following    matrix,       rank    the    blocks    from    best    to 
worst       CI      to      16) .  The      lowest      numbered      block       is      the 

intersection  of  the  best  of  the  rows  and  columns.  The  number 
two  block  is  next  best,  and  so  on.  Note  the  arrows  and  the 
phrases  associated  with  them.  Design  means  the  design  of  the 
layout      or    arrangement.  Do    not    think    of    the    layout    of      one 

workstation,  such  as  the  WLR-1  or  SLQ-17,  but  of  the  entire 
EW  Module.  Think  of  the  scenario  already  presented  in  order 
to      properly    consider    the    workload.  Ask    any    questions      you 

want    or    talk    among    yourselves    or    go    and    look    at    the    layout. 

Multiple  Tasks 
Integrated 

Layout  Enhances 

Specific  Task 
Accoaolishment 

Adequate 
Performance 
Achievable 
(Layout  Sufficient) 

Inadequate 

Performance  due  to 
Layout,  Canr.ot  C:- 
pensate  For  Sub- 
Par  Performance 


cn 
C 
•H 
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o 
u 

S 


w 
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Workload  At     Workloaa  Con-  workload  Biigntly  Workload  hs 

Critical  Level;   siderabiy  Higner  higher  Than  Anticipated; 

Compensation    Than  SnticiDated;  finticioatec;  No  Interference; 

Very  Excessive   EcKpenssticn  Moderate  No  Compensation 

Hi-h;  Irrterferenca  -crnoensation; 

Kajor  v.ir.ir   Interference 


WORKLOAD  IMPROVING 
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LINK  ANALYSIS  QUESTIONNAIRE 
WLR-1  Operator: 

How  many  times  in  an  eight,  hour  watch  do  you: 

1.  Talk/communicate  with  the  SLQ-17  operator? 

2.  View  the  presentation  on  the  SLQ-17  console? 

3.  Talk/communicate  with  the  NTDS  operator? 

4.  View  the  presentation  on  the  NTDS  console? 

5.  View  the  NTDS  Status  Board  (SB)? 

6.  View  the  WLR-1  Status  Board? 

7.  Update  the  WLR-1  Status  Board? 

8.  Check  (visually)  the  SLQ-17  computer? 

9.  Reboot,  reset,  or  work  with  the  SLQ-17  computer? 

10.  Check  MUTE? 

11.  Change  any  settings  on  MUTE? 
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SLQ-17  Operator: 

How  many  times  in  an  eight  hour  watch  do  you: 

1.  Talk/communicate  with  the  NTDS  operator? 

2.  View  the  presentation  on  the  NTDS  console? 

3.  Talk/communicate  with  the  WLR-1  operator? 

4.  View  the  presentation  on  the  WLR-1  console? 

5.  View  the  WLR-1  Status  Board? 

6.  View  the  NTDS  Status  Board? 

7.  Update  the  NTDS  Status  Board? 

8.  Check  (visually)  the  SLQ-17  computer? 

9.  Reboot,  reset,  or  work  with  the  SLQ-17  computer 

10.  Check  NUTE? 

11.  Change  any  settings  on  MUTE? 
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NTDS  Operator/EW  Supervisor: 

How  many  times  in  an  eight  hour  watch  do  you: 

1.  Talk/communicate  with  the  SLQ-17  operator? 

2.  View  the  presentation  on  the  SLQ-17  console? 

3.  Talk/communicate  with  the  WLR-1  operator? 

4.  View  the  presentation  on  the  WLR-1  console? 

5.  View  the  WLR-1  Status  Board? 

6.  View  the  NTDS  Status  Board? 

7.  Update  the  NTDS  Status  Board? 

8.  Check  (visually)  the  SLQ-17  computer? 

9.  Reboot,  reset,  or  work  with  the  SLQ-17  computer? 

10.  Check  MUTE? 

11.  Change  any  settings  on  MUTE? 

12.  Communicate  outside  the  Module? 
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APPENDIX    B 

QUESTIONNAIRE    RESULTS 
Rating    Matrix    Cell    Rank    Order 
In      the    following    matrix,       the    blocks    rank    from    best      to 
worst       (1       to       16).  The       lowest      numbered       block       is       the 

intersection  of  the  best  of  the  rows  and  columns.  The  number 
two  block  is  next  best,  and  so  on.  Note  the  arrows  and  the 
phrases  associated  with  them.  Design  means  the  design  of  the 
layout    or    arrangement. 


Multiple  Tasks 
Integrated 


Layout  Enhances 
Specific  Task 
Accomplishment 

Adequate 
Performance 
Achievable 
(Layout  Sufficient) 

Inadequate 
Performance  cue  to 
Layout,  Cannot  Com- 
pensate For  Sup- 
Par  Performance 


12 


14 


16 


11 


15 


10 


13 


Workload  fit     Workload  Con-  workload  Slightly  Workload  3s 

Critical  i_evei;   sicerably  Higher  Higner  Than  Anticipated; 

Compensation    "^an  Anticipated;  Anticipated;  No  Interference; 

very  Excessive   Compensation  Moderate  No  Compensation 

High;  Interference  Compensation; 

tfajor  *inor  Interference 
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RESULTANT    INTERVAL    SCALE    for    OPERABILITY 


Multiple  Tasks 
Integrated 


54 


75 


88 


in 


Layout  Enhances 
Specific  Task 
Accomplishment 


40 


61 


74 


as 


Adequate 
Perforsance 
Achievable 
(Layout  Sufficient) 

Inadequate 
Performance  due  to 
Layout,  Cannot  Com- 
pensate  For  Sub- 
Par  Performance 


24 


45 


58 


78 


21 


34 


46 


Worxioad  At 
Critical  Level; 
Compensation 
Very  Excessive 


Workload  Con- 
sideraoly  Higher 
Than  Anticipated; 
Compensation 
Hign;  Interference 
Major 


Workload  Slightly 
Higher  Than 
Anticipated; 
Moderate 
Compensation; 
Finer  Interference 


Workload  As 
Anticipated; 

Ho   Interference; 
No  Compensation 
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RESULTS  OF  ALL  OPERATORS 
NOTES:  The  following  results  are  the  Operability  Value  of 
all  reaponaea  for  each  aubtaak.  Generally,  n  equalled  7  for 
the  subtasks,  although  there  were  some  with  only  six 
reaponaea.  The  Standard  Deviation  ia  that  of  the  sample  and 
not  the  population  (i.e.,  the  atandard  deviation  was 
calculated  using  n  vice  n-1).  The  Interval  Scale  shown  was 
baaed  on  the  Ranking  Scale  which  the  EW  operators  provided 
through  the  questionnaire.  The  Ranking  (an  ordinal)  Scale 
was  then  converted  to  an  interval  scale  by  means  of  the 
Delta  Method  <a  linear  expansion) .  The  Criticality  listed 
is  a  mean  for  the  aubtaak.  The  Weighted  Operability  Value 
is  simply  the  Operability  Value  weighted  (multiplied)  by  the 
Criticality. 

EW  Supervisor 


Task:  1.1  Direct  ESM  search 
Subtasks 

1.1.1  Assign  search  parameters  to  SLQ-17 

Operability  Value:  80.143 

Criticality:  3.167 

Weighted  Operability  Value:  253.81 

Weighted  Deficit  Value:  62.89 

1.1.2  Assign  search  parameters  to  WLR-1 

Operability  Value:  49.714 

Criticality:  4.143 

Weighted  Operability  Value:  205.97 

Weighted  Deficit  Value:  208.33 
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1.1.3  Assign  ESH  sensor  report,  responsibilities   --  own  ship 

Operability  Value:  44.000 

Criticality:  4.500 

Weighted  Operability  Value:  198.00 

Weighted  Deficit  Value:  252.00 

1.1.4  Assign  ESM  sensor  report  responsibilities   --  force 

Operability  Value:  24.429 

Criticality:  4.000 

Weighted  Operability  Value:  97.72 

Weighted  Deficit  Value:  302.28 

1.1.5  Initiate  manual  ID  request  -  ship 

Operability  Value:  40.800 

Criticality:  4.250 

Weighted  Operability  Value:  173.40 

Weighted  Deficit  Value:  251.60 

1.1.6  Initiate  manual  ID  request  -  force 

Operability  Value:  22.000 

Criticality:  4.000 

Weighted  Operability  Value:  88.00 

Weighted  Deficit  Value:  312.00 

1.1.7  Monitor  automatic  correlations/associations,  (SLQ-17) 

Operability  Value:  90.333 

Criticality:  3.667 

Weighted  Operability  Value:  331.25 

Weighted  Deficit  Value:  35.45 

Task :  1.2  Report/Disseminate  EW  Information 
Subtaaka 

1.2.1  Report  evaluated  EW  information 

Operability  Value:  3.000 

Criticality:  5.000 

Weighted  Operability  Value:  15.00 

Weighted  Deficit  Value:  485.00 

1.2.2  Provide  EW  recommendations 

Operability  Value:  3.000 

Criticality:  5.000 

Weighted  Operability  Value:  15.000 

Weighted  Deficit  Value:  485.00 
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1.2.3  Update  status  board  near  NTDS  console 

Operability  Value:  60.143 

Criticality:  3.714 

Weighted  Operability  Value:  223.37 

Weighted  Deficit  Value:  148.03 

1.2.4  Brief/debrief  embarked  Airwings 

Operability  Value:  29.286 

Criticality:  3.286 

Weighted  Operability  Value:  96.23 

Weighted  Deficit  Value:  232.37 

1.2.5  Navigation  by  passive  EW 

Operability  Value:  17.500 

Criticality:  3.500 

Weighted  Operability  Value:  61.25 

Weighted  Deficit  Value:  288.75 

Task:  1.3  Counter  Hostile  Environment 

Subtaaka 

1.3.1  Promulgation  of  ECM  employment  criteria 

Operability  Value:  52.500 

Criticality:  4.500 

Weighted  Operability  Value:  236.25 

Weighted  Deficit  Value:  213.75 


NTDS  Operator 

Task:  2.1  Collect  and  enter  EW  data  into  NTDS 
Subtaaka 

2.1.1  Enter  manual  ID  information  into  NTDS 

Operability  Value:  9.857 

Criticality:  4.500 

Weighted  Operability  Value:  45.06 

Weighted  Deficit  Value:  405.64 

2.1.2  Enter  manual  ESM/NTDS  track  associations 

Operability  Value:  20.286 

Criticality:  4.714 

Weighted  Operability  Value:  95.63 

Weighted  Deficit  Value:  375.77 
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2.1.3  Perform  triangulation  of  ESM  bearing  lines 

Operability  Value:  25.000 

Criticality:  4.429 

Weighted  Operability  Value:  110.73 

Weighted  Deficit  Value:  332. IS 

2.1.4  Enter  EW  fixes 

Operability  Value:  22.143 

Criticality:  5.000 

Weighted  Operability  Value:  110.72 

Weighted  Deficit  Value:  389.29 

2.1.5  Advise  operators  of  bearing  resolution 

Operability  Value:  23.714 

Criticality:  4.429 

Weighted  Operability  Value:  105.03 

Weighted  Deficit  Value:  337.87 

2.1.6  Evaluate  externally  reported  ESM  bearings 

Operability  Value:  57.143 

Criticality:  4.286 

Weighted  Operability  Value:  244.91 

Weighted  Deficit  Value:  183.69 


Task:  2.2  Report/Disseminate  EW  Information 
Subtask 

2.2.1  Report  evaluated  EW  information 

Operability  Value:  34.714 

Criticality:  4.571 

Weighted  Operability  Value:  158.68 

Weighted  Deficit  Value:  298.42 

2.2.2  Update  status  board  near  console 

Operability  Value:  51.714 

Criticality:  3.571 

Weighted  Operability  Value:  184.67 

Weighted  Deficit  Value:  172.43 

SLQ-17  Operator 

Task:  3.1  Conduct  ESM  Search 
Subtask 


96 


3.1.1  Monitor  automatic  correlations/associations 

Operability  Value:  52.000 

Criticality:  4.286 

Weighted  Operability  Value:  222.87 

Weighted  Deficit  Value:  205.73 

3.1.2  Establish  operating  modes  of  SLQ-17  <ESM) 

Operability  Value:  80.571 

Criticality:  3.857 

Weighted  Operability  Value:  310.76 

Weighted  Deficit  Value:  74.94 

3.1.3  Enter  detection  and  response  parameters  (ESM/ECM) 

Operability  Value:  45.286 

Criticality:  4.429 

Weighted  Operability  Value:  200.57 

Weighted  Deficit  Value:  242.33 

3.1.4  Monitor  environment  on  NTDS  console 

Operability  Value:  88.143 

Criticality:  4.000 

Weighted  Operability  Value:  352.57 

Weighted  Deficit  Value:  47.43 

3.1.5  Evaluate  displayed  data 

Operability  Value:  46.000 

Criticality:  4.571 

Weighted  Operability  Value:  210.27 

Weighted  Deficit  Value:  246.83 


Task:  3.2  Report/Disseminate  EW  Information 
Subtaak 

3.2.1  Report  evaluated  EW  information 

Operability  Value:  66.429 

Criticality:  4.286 

Weighted  Operability  Value:  284.71 

Weighted  Deficit  Value:  143.89 

3.2.2  Provide  ESM/ECM  data  to  NTDS 

Operability  Value:  67.429 

Criticality:  4.000 

Weighted  Operability  Value:  269.72 

Weighted  Deficit  Value:  130.28 
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3.2.3  Monitor  entry  of  EW  data  into  NTDS 

Operability  Value:  78.571 

Criticality:  2.571 

Weighted  Operability  Value:  202.00 

Weighted  Deficit  Value:  55.09 

3.2.4  Update  status  board 

Operability  Value:  24.857 

Criticality:  3.571 

Weighted  Operability  Value:  88.76 

Weighted  Deficit  Value:  268.34 


Task :  3.3  Counter  Hostile  Environment 
Subtask 

3.3.1  Engage  targets  with  ECM 

Operability  Value:  88.857 

Criticality:  4.714 

Weighted  Operability  Value:  418.87 

Weighted  Deficit  Value:  52.53 

3.3.2  Establish  ECM  operating  Modes 

Operability  Value:  71.714 

Criticality:  4.571 

Weighted  Operability  Value:  327.80 

Weighted  Deficit  Value:  129.30 

3.3.3  Assist  in  promulgation  of  ECM  employment  criteria 

Operability  Value:  82.286 

Criticality:  3.333 

Weighted  Operability  Value:  274.26 

Weighted  Deficit  Value:  59.04 


WLR-1  Operator 

Task:  4.1  Conduct  ESM  Search 

Subtasks 

4.1.1  Search  assigned  bands 

Operability  Value:  3.000 

Criticality:  5.000 

Weighted  Operability  Value:  15.00 

Weighted  Deficit  Value:  485.00 
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4.1.2  Analyze  intercepted  signals 

Operability  Value:  14.286 

Criticality:  5.000 

Weighted  Operability  Value:  71.43 

Weighted  Deficit  Value:  428.57 

4.1.3  Check  intercepts  for  images/harmonics 

Operability  Value:  87.714 

Criticality:  2.286 

Weighted  Operability  Value:  200.51 

Weighted  Deficit  Value:  28.09 

4.1.4  Accurately  DF  intercepted  signals 

Operability  Value:  11.286 

Criticality:  5.000 

Weighted  Operability  Value:  56.43 

Weighted  Deficit  Value:  443.57 

4.1.5  Assist  in  evaluating  ECM 

Operability  Value:  48.571 

Criticality:  2.286 

Weighted  Operability  Value:  111.03 

Weighted  Deficit  Value:  117.57 


Task :  4.2  Report/Disseminate  EW  Information 
Subtasks 

4.2.1  Provide  ESM  data  to  NTDS 

Operability  Value:  3.000 

Criticality:  4.714 

Weighted  Operability  Value:  14.14 

Weighted  Deficit  Value:  457.26 

4.2.2  Report  evaluated  EW  information 

Operability  Value:  7.857 

Criticality:  5.000 

Weighted  Operability  Value:  39.29 

Weighted  Deficit  Value:  460.72 

4.2.3  Update  status  board  near  position 

Operability  Value:  82.286 

Criticality:  2.429 

Weighted  Operability  Value:  199.87 

Weighted  Deficit  Value:  43.03 
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4.2.4  Log  all  intercepts 

Operability  Value:  13.429 

Criticality:  4.571 

Weighted  Operability  Value:  61.38 

Weighted  Deficit  Value:  395.72 


Task:  4.3  EMCON 
Subtaaka 

4.3.1  Monitor  EMCON 

Operability  Value:  46.429 

Criticality:  4.286 

Weighted  Operability  Value:  198.99 

Weighted  Deficit  Value:  229.61 

4.3.2  Report  violations  of  EMCON 

Operability  Value:  44.143 

Criticality:  3.857 

Weighted  Operability  Value:  170.26 

Weighted  Deficit  Value:  215.44 

4.3.3  Log  violations  of  EMCON 

Operability  Value:  49.286 

Criticality:  3.000 

Weighted  Operability  Value:  147.86 

Weighted  Deficit  Value:  152.14 

4.3.4  Monitor  MUTE 

Operability  Value:  35.571 

Criticality:  3.714 

Weighted  Operability  Value:  132.11 

Weighted  Deficit  Value:  239.29 


EW  Status  Board 


Task:  5.1  Maintain  Status  Boards 

Subtaaka 

5.1.1  Communicate  with  operators 

Operability  Value:  10.573 

Criticality:  4.571 

Weighted  Operability  Value:  48.33 

Weighted  Deficit  Value:  408.77 
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5.1.2  Advise  operators  of  any  information  received 

Operability  Value:  24.429 

Criticality:  4.000 

Weighted  Operability  Value:  97.72 

Weighted  Deficit  Value:  302.28 

5.1.3  Update  status  boards 

Operability  Value:  20.289 

Criticality:  4.714 

Weighted  Operability  Value:  95.64 

Weighted  Deficit  Value::  375.76 
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APPENDIX  C 

THE  DELTA  METHOD 
This  appendix  gives  a  brief  description  of  the  delta 
method,  an  algorithm  for  converting  ordinal  measures  on  the 
cells  of  a  matrix  to  a  scale  with  interval  properties 
satisfying  the  conditions  of  additive  conjoint  measurement. 
This  appendix  is  essentially  the  same  as  that  in  Ref .  2, 
only  being  changed  to  reflect  the  present  application.  For 
a  further  and  fuller  description,  see  Coombs  CRef.  63.  The 
method  will  be  briefly  described  using  the  matrix  in  Figure 
C-l.  This  matrix  is  similar  in  form  to  the  4x4  matrix  used 
in  the  OW/SE  rating  matrix  developed  for  the  Subtask 
Questionnaire  but  smaller  in  size. 


Factor  II    Q 


1                   6 

3 

9                       1 

1                   3 

4 

7                       1 

1                   1 

2 

5                      1 

A  B 

Factor  I 
Figure  C-l 
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In    Figure   C-l,    Factors   I   and   II   represent   two 
independent   measures,   and  the  numbers  in  the  cells  of   the 
matrix    represent    an   empirical    ordering    of    overall 
performance   over  combinations  of  factors  I  and  II.    Higher 
numbers  represent  better  overall  performance. 

The  resulting  scales  will  be  interval  measures  of  I  and 
II  as  well  as  overall  performance.  Because  the  scale  is 
additive,  the  measure  of  overall  performance  of  any  cell 
must  be  the  sum  of  the  corresponding  row  and  column  scale 
values.  Furthermore,  the  resulting  performance  measure  mu3t 
reflect  the  ordering  of  the  cells  of  the  matrix. 
Consequently,  any  set  of  scale  values  which  provide  an 
additive  representation  for  a  matrix  must  simultaneously 
satisfy  the  equations  implied  by  the  additive  representation 
and  the  inequalities  specified  by  the  rank  ordering  of  the 
cells  of  the  matrix.  Conditions  under  which  a  set  of  linear 
equations  and  inequalities  have  a  common  solution  are 
specified  mathematically  by  the  Theorem  of  the  Alternative. 
In  practice,  solutions  may  be  found  by  using  various  linear 
programming  techniques.  The  delta  method  is  one  such 
technique  that  is  simple  enough  to  be  done  by  hand  for  small 
matrices . 

The  delta  method  proceeds,  in  general,  as  follows. 
Cells  in  the  matrix  are  initially  given  arbitrary  positive 
scale  values  (represented  by  the  Greek  letter,  delta;  hence, 
the  name) (we  will  replace  delta  with  the  Roman  letter  d,  for 
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ease  of  computation)  which  satisfy  the  equations  specified 
by  the  additive  representation.  For  example,  if  B  were 
assigned  the  value  di  and  Q  were  assigned  the  value  d3,  then 
the  cell  BQ  would  have  the  value  d-i  ♦  d3-  After  initial 
assignments  are  made,  the  relationships  between  scale  values 
and  the  d's  are  changed  to  take  into  account  the  constraints 
given  by  the  rank  ordering  of  the  matrix.  When  the 
procedure  is  completed  and  a  solution  is  found,  scale  values 
are  represented  by  positive  linear  combinations  of  the  d's, 
such  that  any  choice  of  positive  d's  will  lead  to  scale 
values  which  satisfy  both  the  equations  implied  by 
additivity  and  the  inequalities  implied  by  rank  order. 

The  levels  of  each  factor  are  assigned  values  which 
reflect  the  ordering  on  that  factor.  Thus  for  Factor  I,  A 
will  be  assigned  a  value  of  0  since  it  is  the  lowest  level 
of  the  factor.  B  will  be  assigned  an  arbitrary  positive 
value  of  di.  Since  C  is  a  higher  level  of  performance  than 
B,  it  may  be  expressed  as  d]_  +  d2  where  d2  is  an  arbitrary 
positive  constant  representing  the  difference  between  the 
scale  values  of  B  and  C.  Similarly,  in  Factor  II,  P,  G,  and 
R  may  be  assigned  values  of  0,  d3,  and  d4 ,  respectively. 
Since  the  overall  performance  is  an  additive  combination  of 
the  two  factors,  the  scale  value  of  individual  cells  may  be 
stated  in  terms  of  the  d's  as  has  been  done  in  Figure  C-2. 
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d3+d4  R 


Factor 
II 


d3  Q 


0  P 


1    d3-»-d4 

dl*d3+d4 

di+d2+d3+d4   1 

1     d3 

di+d3 

difd2*d3      1 

1      0 

dl 

di+d2       1 

A  B 

0  di 

Factor  I 
Figure  C-2 


di«-d2 


Alternately,  the  scale  values  may  be  displayed  as  in 
Figure  C-3.  this  figure  represents  the  work  sheet  which 
will  be  used  in  the  algorithm.  On  the  top  half  of  the 
figure  are  the  individual  factor  scale  values.  On  the 
bottom  half  are  the  values  of  overall  performance,  listed  in 
decreasing  rank  order.  The  columns  represent  the  four  d's. 
the  numbers  in  the  work  sheet  are  the  coefficients  in  the 
equation : 

Scale  Value  =  coefi  x  di  +  coef2  x  d2  +  coef3  x  d3  ♦  coef4  x 
d4 . 
A  blank  indicates  a  coefficient  of  zero. 
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A 
B 
C 
P 
Q 
R 

CR 

BR 

CO 
AR 
CP 
BQ 
AQ 
BP 
AP 


di    d?    dr^    d4 

111      I      I      I      I      I 

11    I  1    I      1      1      1      1 

1      1      111      1      1      1 

1      1      11    11    1      1      1 

11    11    11    11    1      1      1 

11    1      11    11    1      1      1 

11    11    11    1      1      1      l 

1      1      1  1    1  1    1      1      1 

11    11    1      1      1      1      1 

111      111      1      1      1 

l      1      111      1      1      1 

111      1      1      1      1      1 

Figure  C-3 

The  general  procedure  involves  comparing  cells  adjacent 
in  the  rank  ordering  of  the  matrix  and  redefining  the 
relationship  between  the  scale  values  and  the  d's  so  that 
the  order  of  the  cells  will  be  preserved  for  any  choice  of 
positive  d's.  the  cells  may  be  examined  in  any  order;  in 
this  example  and  the  Layout  Effectiveness  application,  we 
will  start  from  the  lowest  performance  and  proceed  to  the 
highest  level.  This  involves  moving  from  the  bottom  to  the 
top  of  the  work  sheet. 

The  first  pair  of  cells  in  BP  and  AP .  examination  of 
Figure   C-3  shows  that  BP  is  higher  than  AP.    Since  BP   has 
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value  di  and  AP  has  value  O,  it  may  be  concluded  that  di  > 
0.  This  inequality  is  clearly  satisfied  for  any  positive 
value  for  di,  so  it  is  not  necessary  to  redefine  this  value. 
It  will  be  recalled  that  values  were  assigned  to  A  and  B  so 
that  B  would  have  a  higher  value  than  A. 

The  second  inequality  is  AQ  >  BP .  Substituting  the 
values  in  this  inequality  gives  d3  >  di.  This  inequality  is 
not  true  for  all  values  of  d3  and  dj. .  However,  since  d3  > 
di,  it  possible  to  replace  d3  by  di  ♦  d3' ,  for  positive  d3' . 
Now,  for  any  choice  of  positive  di  and  d3' ,  the  inequality 
di  +  d3'  >  di  holds.  On  the  work  sheet,  d3  is  replaced  by 
di  +  d3' ;  that  is,  in  any  row  with  a  d3,  a  di  and  a  d3'  are 
added  and  the  d3  is  deleted.  For  convenience,  and  because 
the  d3'  column  looks  exactly  like  the  d3  column,  the  d3' 
column  is  put  where  d3  was.  This  is  merely  a  relabeling  of 
columns.  Note  that  as  many  "marks'  as  were  in  the  d3  column 
are  added  to  the  di  column.  The  work  sheet  at  this  point 
looks  like  Figure  C-4. 

The  next  inequality,  BQ  >  AQ  implies  2di  ♦  d3'  >  di  + 
d3'  or  di  >  0.  Again  it  is  not  necessary  to  make  any 
changes  to  satisfy  this  constant. 
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A 

B 
C 
P 
Q 
R 

CR 

BR 

CQ 

AR 

CP 

BQ 

AQ 

BP 

AP 


di    d?    dp'  da. 

I      I      I      I      I      I      I 

111      I      I      I      I      I 

II    II    I      I      I      I      I 

I      I      I      I      I      I      I 

111      111      I      I      I 

II    I      II    II    I      I      I 

111   I  1    11    11    1      1      1 

1  1  1  1  1  1  1 
111   1      11    11    1      1      1 

1  1  1  1  1  1  1 
111   11    11    1      1      1      1 

I  1      1      1      1      1      1 

II  1      11    11    1      1      1 

1  1  1  1  1  1  1 
1  1    1  1    1      1      1      1      1 

1      1      1      1      1      1      1 

1 11   1      111      1      1      1 

1  1  1  1  1  1  1 
111      111      1      1      1 

1  1  1  1  1  1  1 
111      1      1      1      1      1 

1  1  1  1  1  1  1 
1      1      1      l      1      1      1 

AQ>BP=d3>di 

»d3=di*d3' 
BP>AP=di>0 

do  nothing 


Figure  C-4 

Proceeding  up  the  work  sheet,  the  next  inequality  is  CP 
>  BQ .  This  implies  that  d2  >  di  +  d3' .  Here,  we  do  the 
same  as  before,  in  that  we  redefine  d2  as  di  +  d3'  +  d2' » 
and  consequently  replacing  d2  in  every  row  in  which  it 
occurs  by  a  di,  d3' ,  and  d2' •  Again  d2'  is  put  in  the 
column  where  d2  was.  Consequently,  the  procedure  involves 
relabeling  the  d2  column  d2'  and  adding  a  di  and  a  d3'  for 
each  d2  in  any  row  in  which  a  d2  appears.  The  work  sheet  at 
this  point  looks  like  Figure  C-5. 
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A 

B 
C 
P 
Q 
R 

CR 

BR 

CQ 

AR 

CP 

BQ 

AQ 

BP 

AP 


di          d-?'       d-*'       da 

I             I             I             I             I             I             I 

II          I             I             I             I             I             I 

I  11       I  1          11          1             1             1             1 

1             1             1             1             1             1             1 

111             111             1             1             1 

11          1             11          11          1             1             1 

1111     11          111       11          1             1             1 

1  1  1  1  1  1  1 
111       1             11          11          1             1             1 

1  1  1  1  1  1  1 
1111     11          111       (III 

I  1             1             1             1             1             1 

II  1             11          11          1             1             l 

1             1             1             1             1             1             1 

inn       ii       i          i          i          i 

i  i  i  i  i  i  i 
i  n      i          hi          i          i          i 

i  i  i  i  i  i  i 
iii          hi          i          i          i 

i  i  i  i  i  i  i 
hi          i          i          i          i          i 

i          i          i          i          i          i          i 

i          i          i          i          i          i          i 

CP>Ba=d2>di-d3' 
d2=di+d3' +d2' 
BQ>AQ=di>0 

do  nothing. 
AQ>BP=d3>di 

d3=di+d3 ' 
BP>AP=di>0 

do  nothing. 


Figure  C-5 

The  inequality  AR  >  CP  implies  a  similar  inequality 
among  d's  and  is  handled  the  same  way.  The  work  sheet  at 
this  point  is  given  in  Figure  C-6. 

The  ordering  of  CQ  >  AR  implies  an  inequality  among  tne 
d's  which  is  handled  somewhat  differently  from  the  previous 
inequalities.  CQ  >  AR  implies  that  di  *  d3'  >  d4 ' .  Since 
there  are  two  d's  on  the  left  side  of  the  inequality,  it  is 
not  possible  simply  to  make  the  substitution: 

di  -  d3'  =  da/  ♦  di'  +  d3'  '  . 
Some  rows  may  have  different  numbers  of  di  and  d3',   so  this 
replacement  rule  would  be  impossible  to  implement. 
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The  following  three  step  method  is  used  to  redefine  the 
d's.  First,  d4'  is  split  into  two  parts,  d4' '  and  d5 . 
Since  di  +  d3'  >  d4' ,  this  division  may  be  arbitrarily  done 
so  that  di  >  d4"'  and  d3'  >  ds-  The  preceding  two 
inequalities  may  be  handled  by 


A 
B 
C 
P 
Q 
R 

CR 

BR 

CO 

AR 

CP 

BQ 

AQ 

BP 

AP 


di    d?'   d^'   da' 

I      I      I      I      I      I      I 

111      I      I      I      I      I 

I  11   11    11    1      1      1      1 

1      1      1      1      1      1      l 

111      111      1      1      1 

1 11   1 1    11    11    1      1      1 

1 1111 1 11   1  11   1  1    1      1      1 

1  1  1  1  1  1  1 
1  111  11    11    11    1      l      l 

1      1      1      1      1      l      1 

1  111  1 1    l  11   1      l      1      l 

1  1  1  1  1  l  1 
1 11   I  1    11    11    l      1      l 

1  l  1  l  1  1  l 
1 11   l 1    11    1      I      I      I 

1  1  1  1  1  1  1 
1111      111      1      1      l 

1  1  1  l  1  1  1 
111      111      1      1      i 

1  1  I  1  1  1  1 
111      1      1      1      i      1 

1  1  1  1  1  1  1 
1      1      1      l      1      1      I 

AR>CP  =  d4>di  -fd2' 

d4=di +d2' ♦04' 
CP>BQ=d2>di-d3' 

d2=di *d3' +d2 ' 
BQ>AQ=di>0 

do  nothing . 
AQ>BP=d3>di 

d3=di+d3' 
BP>AP=di>0 

do  nothinq . 


Figure  C-6 
previously   discussed  methods.    Thus  the  three  replacements 


are 


d4'  =  d4"  +  ds, 

di  =  d4''  +  di',  and 

d3'  =  d3"  ♦  d5. 
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Now  for  any  choice  of  positive  d's,  CQ  >  AR  is 
satisfied.   The  work  sheet  now  looks  like  Figure  C-7. 

The  methods  of  handling  the  remaining  inequalities  have 
already  been  discussed.  Note  that  the  steps  to  complete  the 
worksheet  is  (matrix  size  -  1),  or  in  the  example,  8.  For 
the  Module  Operability  application  there  were  15  steps. 

When  completed,  the  top  half  of  the  work  sheet  shows  the 

relationship  between  scale  values  and  the  newly  defined  d's. 

For  the  example  given,  the  following  relationships  hold: 

A  =  0 

B  =  di'  ♦  d3"  ♦  d4"' 

C  =  2di'  ♦  d2'  +  3d3"  -  2d4'"  +  ds 

P  =  0 

Q  =  di'  +  2d3"  ♦  d4'"  +  d5 

R  =  2di'  ♦  d2'  ♦  4d3"  +  3d4"  '  ♦  2d5 


A 
B 
C 
P 
Q 
R 

CR 

BR 

CQ 

AR 

CP 

BQ 

AQ 

BP 

AP 


di    d?'   dq'   da'   ds 

l      I      I      I      l      I      i 

111      I      I      I      I      l 

I  11   I  1     11    1  11   1  1     1      i 

1      1      1      1      1      1      l 

11    1      11    11    11     1      i 

111   11     11     1  111  1 11   1      l 

llllllll    111    111    15     1111  1 

1  1  1  1  1  1  1 
1  111  11     11     1 1111 1 11    1      1 

1  1  1  1  1  1  I 
1  111  11     111    1  111  1111      1 

1  I  1  1  1  1  1 
111    11     11     1  111  1111      i 

1  1  1  1  1  1  1 
1 11    1 1     11     1  11    1  1     1      1 

I  1  1  1  1  1  1 
1111      11     1  11    1  1     1      I 

I  1      1      1      1      1      1 

II  1      11     11     11     1      I 

1  1  1  1  1  1  1 
111      1      111      1      1 

1  1  1  1  1  1  1 
1      1      1      1      1      1      1 

Cd4'=d4'  '  -^5 

(di=d4"-di' 
CQ>AR=di+d3' >d4' 

td3'=d5-d3' ' 
AR>CP=d4>di+d2' 

d4=di +d2 ' *d4 ' 
CP>BQ=d2>di*d3' 

d2=di +d3' +d2 ' 
BQ>AQ=di>0 

do  nothing. 
AQ>BP=d3>di 

d3=di *d3 ' 
BP>AP=di>0 

do  nothing. 


Figure  C-7 
111 


Any  choice  of  positive  d's  will  give  a  scale  for  Factors  I 
and  II  as  well  as  overall  performance  which  is  an  additive 
representation,  and  in  which  the  overall  ordering  agrees 
with  the  empirical  ordering.  Figure  C-8  contains  the 
results  of  this  example. 


djj    do' 


da: 


d«a'"   d^ 


dft 


d7_ 


dR. 


A  J_ 
B  I  1 


I  1 


C  I  11 
P  I 


I  1 


1111   111 


I  1 


I 


Q  I  1 
R  111 


I  11 


I  1 


I  1 


1111  1111   111 


CR  11111  111 


1111  1111 


BR  I  111   I  1 


15 


11111  111 


CQ  I  111   I  1 


15 


111   111 


AR  I  11 


I  1 


1111  1111 


11 


CP  I  11 


I  1 


I  111   I  11 


I  1 


BQ  I  11 
AQ  II 


1111   111 


I  1 


I  11 


I  1 


BP  J_L 
AP  I 


I  1 


I 


I 


I 


=  0 

=  3 

=  9 

=  0 

=  5 

=  12 

=  20 

=  15 
=  14 
=  12 
=  9 
=  S 
=  5 
3 
0 


Figure  C-S 

A  common  choice  of  d's  is  to  make  them  all  equal  to  1. 
This  choice  for  d  yields  a  set  of  scale  values  which 
represent  the  set  of  minimal  integers  which  will  produce  the 
requires  rank  order  of  the  matrix  cells. 

The  completed  work  sheet  for  the  Module  Operability 
application  follows.  Note  that  the  numbers  within  the  d's 
are  expressed  in  arabic  numerals  for  viewing  ease  and  the 
totals  are  listed  on  the  side. 


112 


di  " 

"  <±->" 

'"   dV 

<±A' 

"  ds' 

"  dfi' 

"   d7 

dA 

A 

1 

1 

1 

1 

1 

1 

1 

1       1 

B 

12 

13 

14 

13 

13 

1  1 

1  4 

12      1 

G 

13 

15 

17 

15 

14 

12 

16 

13      1 

D 

14 

16 

1  10 

17 

16 

12 
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